WRITING YOUR OWN SMTP GATEWAY
by John Rodriguez (john-r@vbonline.com)
November, 1995
![[MJA Accounting]](/vb-mag/images/banmjaac.gif)
Author's note:
There is a sample project that comes with this. This code is more for
understanding how Domain Name Resolution works. The module DNS.BAS is what will be used in
the final code project that will accompany the next and final article. Also, I have received
some requests for assistance in constructing projects. I apologize to those for not getting
back as quickly as possible as I have been very busy. If you are still in need of
assistance, please send me an e-mail message and I will try to get back to you as soon as I
can.
When I first began writing my SMTP gateway over a year ago, I was very new to the Internet. Oh, I have heard all of the buzz words and understood the concepts alright, but in reality, I didn't have a clue as to how many things that are available as network services. In this capacity, when I completed my initial application, I had some trouble. The trouble started when a user on my system tried to send a message to someone at IBM. Needless to say, my colleague had quite a bit to say about the capability of my gateway. Then, of course, came the boss. He, too, was trying to send a message to someone who should have been a registered host. Then it was my turn to receive failure when I tried to send a message to the Microsoft Developer's Network email address. At this point, things were getting real ugly.. and so, I read a little more and found out that I was missing something very important in SMTP services.. resolving domain names.
For those of you who are new to the Internet, you may be surprised to know how organized the entire network really is. What makes it so chaotic is the perception of just having so many places to go. The order of the system is brought together by a simple, yet powerful feature of the network. This mechanism is called the Domain Name Server or DNS for short. This server does the job of keeping track of the names of all the registered hosts under its .. domain. Whenever you want to find information about a particular host, the DNS can send back vital information on how to access that host and what capabilities it has. Most requests are for resolving host names into IP (internet protocol) addresses for connection.
What does this have to do with SMTP? Well, the answer lies in how the DNS structures its database. The host computer that does the job of resolving domain names keeps its information in a structured database. When a remote computer makes a query about a host, the local computer retrieves information from its database and sends it back to the remote host, usually in the form of a datagram. What an SMTP Gateway does is resolve the domain to an IP address of where the message is supposed to be delivered. Depending on the implementation of the gateway, the name is resolved by either a local host file or by querying a DNS. The local DNS will then check its database for any records relating to the request. If the request cannot be satisfied, then depending on the capability of the name server, the system will either request information from other name servers or return a response back to the remote host indicating either failure or a list of name servers that might have the information the remote system is looking for. Of course, if the name server can find the information then it is returned to the remote system.
Alright, so now armed with this information, we should be able to find out the IP address of our domain. Well, almost. You see, some domains have an IP address. Other domains do not. This is especially true for mail gateways. If you don't believe me, just try and ftp to microsoft.com. You will always get a "host not found" or some other error when your ftp program tries to resolve the name. However, you CAN send mail to someone at microsoft.com. However, if you try to make a direct connection to microsoft.com, you will never make it. But, there is a way to get information on how to get mail to a domain that doesn't exist... or is there?
The DNS has a database of registered hosts. It also has a "cache" of hosts that it had to resolve from requests made by remote hosts. We will discuss this operation in a moment, but first, let's understand how the database is organized.
The database is organized into records. Each record contains information about a particular host. The records themselves are different in that each record for a host contains different information. While this is a more general overview and may not be "technically correct", the dea is to understand how the DNS communicates this information. To understand all of the nuances of the DNS operation is beyond the scope of this article, however, it might be a good idea to check out the DNS operations in RFC's 1034 and 1035. For now, we will concentrate on the specific information we will need for our SMTP Gateway.
Since we know about the database, we need to understand how to retrieve the information from the database. First, though, we need to determine what type or types of records we need to satisfy our requirements. The DNS carries a variety of record types that serve many types of requests. In our implementation, there are two types of records we will use - a Mail Exchange or MX record and a host record or an A record. For SMTP, the MX record is what will solve our problem with trying to connect to hosts like microsoft.com. An MX record points to a registered host that can receive mail for a particular domain, as in this case, microsoft.com. The A record will be necessary if we can't find an MX record for our application. This is because, some hosts can service SMTP without having an MX record and its capabilities are marked in another record called WKS or Well Known Services... although we won't be discussing WKS records. And if there are no A records for our host, you can pretty much bet we aren't going to find the host we need. Of course, there is one other type of record that is rarely used, but does show up unexpectedly sometimes, and that is the Canonical or CNAME record. It really is an alternate name for a host that shares the same IP address.. generally. All you should really know is how to deal with it.
Now that we know a little... well not much anyway, about the DNS, we can now access the DNS for records to tell us where to send our mail. To do this, we are going to need to understand how to initiate a query. Each access of the DNS requires that you send a query for records directly to the DNS. The DNS looks at the query, then checks it database to fill your request. If the database does not have an answer, then it will do one of two things based on the query. It can either do a recursive search, which means it will then query another name server that might be able to satisfy the request, or, it will send you information on other DNS hosts that may have the answer to your query. The functionality is called recursion. Recursion is an option available on most DNS hosts.. however, times being the way they are, some DNS hosts have yet to implement recursion. Since the explosion of the internet in the last few years, more and more software vendors are implementing as many optional requirements in their products. For those that do not implement recursion, the query is processed as normal except for the recursive part - naturally.
The next part is actually constructing the query and then sending it to the host. The query has a specific format as written in RFC 1035. I will not cover the specifics of this as the query and, of course, the sample code provided with this article should do enough to cover the actual construction of the query and the response. What is important is the understanding of the mechanics of how the query is sent. There are two ways to send the query - by TCP connection or by UDP datagram. Either type works, but for our implementation, we will use the UDP datagram. In most cases, a datagram is a preferred method because of its simplicity and ease of use. Once the datagram is assembled properly, the next step is to send it and then wait for a response. Once you have received your response, you must now take apart the datagram and extract the records. The returned datagram is returned in 5 basic parts: the header, the query, the answer records, the authority records, and additional records.
When you disassemble the datagram, the header contains information on how many records are in each section. Once you establish the count, you can then extract each record from the datagram sequentially and then determine your next course of action. Of course, you need to understand what each record contains.
In each section, a resource record has a specific construction. The answer, name server, and additional record sections are constructed basically the same. The query section is the same as you sent to the DNS. It's included as a reference if your implementation can handle more than one request at a time. If you have a number of applications that resolve domain names, the implementation should be able to process more than one request. In our SMTP Gateway, we handle each request completely before performing another query. This means that we do not need to worry about multiple requests at the same time. In the sample project, we only handle one request at a time.
That leaves us with the construction of the other three sections and how the records are constructed. Each record contains the following structure:
The records follow one another based on the counts provided in the reply. All you need to do is extract them in order from the datagram. Once you have done that, you can then process them appropriately.
Oh boy... can you see the monkey wrench falling into machinery? Well, here it is.. you cannot make any assumption about the size of each record in the datagram. The only thing we can be sure of is the datagram itself cannot be longer than 512 bytes. However, the difference in record construction depends on what is returned by the DNS. This can make for some very interesting situations where you may get a lot of records back. The problem resides in the fact because domain names vary in size, it doesn't make sense to establish a finite packet size because each record would have to be the same length. That doesn't do well because the domain name portion of the record would have to be as wide as the largest domain name. Since the largest domain name could be as large as 255 characters, you would have only enough room for about 2 records. Ugh.. what a mess. So, each record holds just enough of the domain name... but what if you had eight records that needed to be returned that had part of its domain name thisisareallylongdomainname.com? If you had eight records that started with say thisishost1 to make thisishost1.thisisareallylongdomainname.com and increment it for each host, most of your datagram would be taken up by large names... so the DNS has to have a method to limit the size of domain names and like values that could be really long.
How the DNS handles this is by compression of domain names.. what? Like PKZIP? No.. not exactly.. but a relatively simple set of pointers that reflect previous references to domain name labels or other similar references. Need to digress here for an explanation of domain labels. Let's take the host microsoft.com - everybody's favorite. A label in this case is the string between the dots. "microsoft" and "com" are labels. When a DNS constructs the datagram for microsoft.com, it does so like this...
The number in front of "microsoft" and "com" tell you how many characters to read starting from the next position. The dot is automatically assumed between labels. When you get to the end, you should get another number or a zero to indicate you got to the end of the domain name. Labels are limited to 63 characters. Now.. how does compression work? Simple. When you come to a number position, you should test the two most significant bits of the byte. If they are both set to 1 (if you use the And operator on the byte with decimal 192, you should get 192) . That means you are at a pointer. The value of the next 14 bits will tell you where in the datagram you need to go for the next label sequence. To do this, And the last 6 bits of the pointer byte (or you can just subtract 192 from the entire ASCII value of the byte), multipy by 256 and then add the value of the next byte. Oops.. wait a minute.. how do you get the value of a "byte" in VB? The datagram comes back as a string. To get the value, all you need to do is find the ASCII value of that position in the string. For example, in our microsoft.com graphic above, the number 9 is actually represented as a tab character (chr$(9)).. all you need is the ASC() function to retrieve the ASCII value and there's your number.
The graphic below illustrates this point...
The key to remember of reading and parsing datagrams is to remember where you left off in the datagram when you come to a pointer. If you lose your place, you could get some real funny looking records!
Normally, when a datagram returns records, the Answer and Authority sections contain the domain names of the resource records and the additional sections contain the IP addresses, if any, associated with the Answer and Authority records. In this regard, as you get closer to the end of the datagram, the more compression takes place.
As was stated we need the MX and A records for handling the SMTP domain name resolution. So what do we do with these records once we get them? Do we need to trash them? Nope. Well.. how long do we keep them? In each record sent, a small field called the Time To Live or TTL indicates to us how long we should keep this record in seconds. Why? Let's suppose that we have a guy who sends mail to microsoft every three minutes. That in itself, isn't bad. Now suppose we send a piece of mail to a different host every 20 seconds. Well.. every twenty seconds is OK.. but suppose you have 1000 other hosts resolving domain names every 20 seconds... uh-oh.. a lot of hits on the DNS are going to take place. That could cause a system administrator great heartburn if the DNS host decides to take a hiatus from processing requests. So the TTL field was added that tells you how long to keep the record. Normally, the TTL is used by the DNS when it does a recursive search for a request so if any subsequent searches are made for the same information, the response time is much quicker. However, the DNS still has to keep processing the request. What our SMTP gateway will do is maintain a cached database of MX and A records of domains so we don't have to hit the DNS until a TTL has expired on a particular record. In high traffic areas, this functionality helps to increase performance dramatically, especially for looking up host information.
Ok.. now the good stuff. The sample project itself is designed to do domain name resolution, along with IP to domain name resolution. What is key is the file DNS.BAS within the project which does all of the actual processing. The information returned is an array of records that satisfies the query. For our eventual use in the SMTP Gateway project, the DNS lookup utility includes provisions for the TTL field so we can maintain our own cache of DNS entries.
The code listing for the code of DNS.BAS which we will use for our SMTP Gateway. This same code is in the sample project but if you're only interested in the DNS.BAS code, then by all means, you can use this code by using your favorite cut and paste technique.. or whatever. This code isn't commented very well because of my time constraints. If you have any problems, you can send me an e-mail message and I will provide any assistance that I can. However, most of you are pretty smart and I think you will find the code pretty easy to follow.
Click here to go back to the November '95 Article Index