Visual Basic Online
WRITING YOUR OWN SMTP GATEWAY
by M. John Rodriguez (john-r@vb-online.com)
September, 1995
So.. you want to write your own SMTP gateway and you aren't sure how to do it? Or maybe you have done it, but want to do cool things like MX Record lookup automatically? What about support for Extended SMTP (ESMTP)? In this series of three articles, I will attempt to go over the fundamentals of developing an SMTP gateway and how you can tailor the model to suit your own needs. I am going to use the code that I developed for my SMTP gateway that I developed for my 26 sites across Europe using Visual Basic and a couple of VBX's to handle the WinSOCK interface.

TCP/IP, WinSOCK and Visual Basic

When I first started this project, I was given wide latitude of development. It had already been decided to use Visual Basic because the boss and I felt comfortable with the capabilities.. not to mention the fact that both of us have done a lot of programming in BASIC. This was a factor that was decided before I even realized I would need to write a gateway. The other part was that my boss knew nothing about C/C++ and so if he were to review my work, he would need to understand everything I was doing. After discussing this with a couple of associates, I was surprised to hear that it could not be done in VB because it was "too slow" or "VB is not a real programming language." Well, the challenge was issued, whether implied or explicit, and I took it on. I had no real programming experience in TCP/IP or even networking for that matter. So this was all uncharted territory. Then I discovered Windows Sockets and the WinSOCK stack and spent many days going over the API, trying to determine how I was to get and receive TCP/IP packets. Almost immediately I began to ask the question: How do I get VB to communicate with the WinSOCK API?

Because of the way the WinSOCK API is, using VB would work in some calls but not in the more critical ones. So, I explored the world of TCP/IP based VBX's. There are a number of vendors that offer a multitude of tools for developing TCP/IP applications but I needed to find one that met the following criteria:

  1. It had to be easy to use without a whole lot of extraneous properties. One thing that really turns me off about VBX's is the overwhelming number of properties some vendors have for them. Sure it can give you a lot of control and flexibility, but also makes it easier to create unintentional errors.
  2. It had to be able to work using any WinSOCK 1.1 stack.
  3. It had to give me the greatest flexibility in development of the application, yet, handle most of the underlying work of the WinSOCK API. In other words, I want some control over what the VBX does without having to understand underlying WinSOCK functionality.
  4. It had to be cost effective. In otherwords, inexpensive to aquire and come with a no-royalty clause for distribution of the final product. It also had to be easy to add updates and be relatively accessible without waiting for it to come in the mail.
Choosing the right VBX In order to evaluate the products available, I downloaded a number from various sources as shareware. One product came with a run-time license fee for each instance of the application. It even had a mechanism built in that could find out if the instance was duplicated across the network! Each instance was about fifty bucks, and I would probably need LOTS of copies.. could easily go into the thousands of dollars. They offered a compile time or OEM license but that was a couple of grand and only lasted for a year if you wanted updates. It did work OK for my initial test and seemed to provide most of the functionality I wanted but it also had a lot of overhead and didn't support multiple listening off the same port. This was a critical feature of the daemon functionality which this control did not support. It also seemed to work differently for different types of WinSOCK stacks.

Another I evaluated implemented SMTP functionality right away. Problem was, I needed to support the ESMTP standard in a future version and there was no guarantee it would work. Then the killer part was the General Protection Faults (GPF) using SuperTCP, which is the primary TCP/IP stack used in my organization. Everytime the VBX did a host lookup the GPF would result. Not good. There were several others. Some were better, but most lacked a requirement or two that I needed for development of my application. Suffice it to say, it was not going well. Then enter DevSoft.

This may seem like a plug for DevSoft because I have gotten the support I needed and the folks there are very nice. They also made a product that, in my opinion is just exactly what I was looking for. For $25.00 per single development license, DevSoft's IPDAEMON, IPPORT and UDPPORT custom controls give you the maximum flexibility in TCP/IP application with just a few logical properties and events. The events are practically the same across each control making it easy to code. Each VBX is small, but powerful. The one feature I liked more than anything was the 32 simultaneous connections of the IPDAEMON control over the same port. Imagine having 32 connections at once to receive mail! Although I have never had more than 3 during busy periods, I have seen it go up to 14.. but only after my machine crashed and I hadn't noticed for a day or so. I didn't lose a single message either. At least, no one said anything.. and believe me, my office generates about 5000 messages a month with most destined to go through the gateway, and they love their e-mail!

DevSoft also makes it easy to download, install and manage updates of their product, placing them in a variety of places on the internet as well as CompuServe and America On-Line. If you already have registered a copy (either by direct purchase from PsL or from CompuServe's SWREG) then to update your copy, just copy the VBX into your WINDOWS\SYSTEM directory and continue on. There are no royalties for distributing the VBX with your application and works over every WinSOCK stack that I have used with no problems whatsoever. Now for the good part. The SMTP Gateway which will be discussed uses the TCP/IP controls from DevSoft. However, you do not have to use them. The model is written in such a way as to let you use any one you choose. However, there is one property that may or may not be available on other controls that the daemon part uses but will be discussed later.

Required Reading
The next step in the development is to read all you can about SMTP and how it works. The recommended reading list for SMTP for me was from the Request For Comments (RFC) documentation. There are a number of RFC's that you will need to read in order to understand all the functionality necessary to write an appropriate gateway for your mail system. There are several places you can download the RFC's from. The one I use is nic.ddn.mil in the /rfc directory. Another one is ds.internic.net. There are others but I have not accessed them and I am not sure where they are. However, you can use anonymous ftp to access most sites that carry RFC's. The following are a list of RFC's I recommend you get and why I think you may need them.

Basic SMTP. A couple of required readings for the SMTP mechanism itself. There are other RFC's that deal with SMTP in other mail enviroments (ie X.400) but are not critical to this implementation. RFC 821 - Simple Mail Transfer Protocol (SMTP). By far the most important since you need this for the gateway. We will use this documentation primarily throughout the articles.
RFC 1047 - Duplicate Messages and SMTP. A mechanism on how to eliminate duplicate SMTP messages. The model follows this recommendation.
RFC 1123 - Requirements for Internet Hosts -- Application and Support. This document refines many of the standards including RFC-821 and how to handle different situations. READ IT! It makes some of the processes a lot easier.

Message Format. I would consider this required reading in that messages need to be in a format that can be sent via SMTP. The articles here explain the structure of the message so that you can identify how a message needs to be sent. RFC 1505 is added for those who do message encoding of attached files and such. It is not paramount to the development, but does have some importance in the mail interface which is discussed later.

RFC 822 - Standard for the Format of ARPA Internet Text Messages.
RFC 1505 - Encoding Header Field for Internet Messages.

Domain Name Server (DNS) Services. The article goes into detail on how to implement a mechanism to find mail hosts. The Mail Exchange (MX) record is very important in determining where mail can be delivered. RFC 974 discusses the implementation, while RFC 1034 and 1035 discuss how the DNS host is implemented, accessed, and how records are formatted.

RFC 974 - Mail Routing and the Domain System.
RFC 1034 - Domain Names - Concepts And Facilities. Critical in the understanding of the DNS host and how records are managed.
RFC 1035- Domain Names - Implementation And Specification. Explains how we will access the DNS host to find resource records (RR's). The MX record is what we are after and our implementation is tailored for that specifically.

Additional Reading: These subjects are provided as general reading and are not directly discussed in this article set. However, some functionality may be necessary in your implementation and these documents represent some of the functionality which can be included. This is not an all inclusive list, but a good starting point.

MIME Support. RFC 1344 - Implications of MIME for Internet Mail Gateways.
RFC 1428 - Transition of Internet Mail from Just-Send-8 to 8bit-SMTP/MIME.
RFC 1521 - MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies.
RFC 1522 - MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text.
RFC 1524 - A User Agent Configuration Mechanism For Multimedia Mail Format Information.
RFC 1556 - Handling of Bi-directional Texts in MIME.
RFC 1563 - The text/enriched MIME Content-type.

Unicode: All I can tell you about this is that it is coming. With the advent of several operating systems adopting Unicode, these two might be worth looking at and understanding. You may or may not have any use for it right now, but it is a good idea to have these around just in case.

RFC 1641 - Using Unicode with MIME.
RFC 1642 - UTF-7: A Mail-Safe Transformation Format of Unicode.

ESMTP. Note: ESMTP is not discussed at all in the article. However, the reference subjects are provided for your information. The gateway model used is designed so you can add ESMTP capability with a minimal amount of code. This is not all inclusive and there are other RFC's that deal with things such as Macintosh files and other types of extensions.

RFC 1651 - SMTP Service Extensions. This is ESMTP.
RFC 1652 - SMTP Service Extension for 8bit-MIMEtransport.
RFC 1653 - SMTP Service Extension for Message Size Declaration.

Other.

RFC 1711 - Classifications in E-mail Routing. I have briefly read this because most of it I already knew or figured out. But it gives a good picture of Internet E-Mail and probably is just good reading.

These RFC's are not all encompassing and at the rate the internet is growing, you can bet these will be changed or updated. Some of these RFC's have undergone revisions in less than a year after initial release.

Preparing the Model.

After you have read all that you need to understand how SMTP works, we need to build the model for our gateway. There are three simple approaches that can be used. Which one really depends on you. First, however, we need to establish a terminology base so the ideas can be effectively communicated. To do this we need to establish what kind of components we need to create. Here are the three components we will develop for our gateway: The SMTP Server (Daemon), The SMTP Client (Mailer), The SMTP Mail Interface (message manager).

At this point we can build any one of three different models of construction. Below are the three models and included are the pro's and con's of each method.

Model 1: Create three separate applications, one for each component.

Pro's:

Con's: Model 2: Create two applications - one for the SMTP functions and one for the Mail Interface.

Pro's:

Con's: Model 3: Combine all components into one application.

Pro's:

Con's Other Considerations in Preparing the Model

Visual Basic: First of all, VB is limited. There are a lot of things that VB cannot do as well as C/C++ tools can. Secondly, VB is slower and can cause some trouble in the event driven environment because of the way the VB system is designed. Having the code run from an interpreter (in this case VBRUNXXX.DLL) does slow the system down. Creating code that is sloppy and not system friendly can kill your product in a heartbeat. Thirdly, development of the product takes patience. SMTP is a rigid protocol that can cause you hours of heartache in testing, so be patient.

Resources: Find out what platform you are going to run the application from. I have tried to tailor this to run on a 386 DX-40 with 4 MB RAM running Windows for Workgroups 3.11 and Microsoft TCP/IP-32. However, it can run with SuperTCP and Trumpet WinSOCK and several others. The only thing I am limited to is what DevSoft or any vendor who supplies VBX's give me. Most of the resources are consumed by the VBX's utilitizing the TCP/IP stack so make sure you consider that. The 386 doesn't do well in high traffic areas but does run the application well enough in low traffic areas.

External Resources: Find out where an SMTP gateway or server is running and send messages back and forth from it. It should be a registered gateway so that you can test your application's functionality. It may not always be possible, but you should be able to. If you are writing this for use directly on the internet, then you will need as much access as possible.. or you could improvise by writing each component and testing it against itself. It is not the best way to test, but sometimes you have to be resourceful.

About the SMTP Gateway

Now we are ready to build the functionality of the gateway. The model I am going to use is Model 2. We are going to build the client and the server into one application and the mail interface into another application. At this point, I will tell you that we will not go into the actual design and development of the mail interface because that is something for you to do. This provides the model with modularity. Now we need to define the components.

Here is the model as it looks in a graphical mode:

(Model Image)

As you can see, the graphic shows internet access from the gateway via the mailer and the server. Each has a separate queue in which to handle messages. The Mail Interface is what moves the messages to and from the normal mail system as well as between the SMTP modules. You can effectively make your SMTP application a relay is you so desire to accomodate messaging between hosts that your gateway may have access to but another host may not.

A brief overview of the Mail Interface

While we will not specifically go into developing the mail interface, there are some things you may consider for your specific gateway.

  1. Relay of internet mail. Do you want to help or not?
  2. Handling of multiple post offices. Do you want to change your post office capability into workgroup functionality?
  3. MIME capabilities. Do you want this part to do any MIME processing?
  4. MAPI, VMS, MHS, POP, IMAP, etc. compatibility. What kind of logistical or development problems are you going to have supporting these types of post office systems? Are you going to handle more than one type?
In our model, we attempt to divide the responsibility of handling messages between different components. The Mail Interface keeps the messages moving from the gateway to the post office. It might be that you have two or three relay components and provide support for different types of post offices. In any case, that part of the development is left up to you.

Conclusion

As I said, this is a three-part series on the development of the SMTP server. This first article is just an overview of the development process and anything you may need to get you started. In the next article, we are going to cover the SMTP Server and how to build it. We will then go over briefly how to do a DNS lookup for MX records. The third article will cover the mailer process, merging the DNS functions and then the final product.

Next month, the sample code for this product will be available here, in Visual Basic Online. Stay tuned.

[About M. John Rodriguez: John is the programmer/developer for the Civilian Personnel Division, United States Army, Europe. He has been in programming for about 12 years and has been involved with his organization's regionalization project, developing an e-mail based database management system for the last year. The entire system is developed in Visual Basic and MS Access and includes an interpersonal mail client, electronic document manager, electronic resume, and of course, the SMTP gateway, all of which he calls the "SMART E-Mail System." He is married with two children and has a dog (doesn't everybody?) In his spare time (whenever it is available), he does a lot of yard work, family outings, and anything to do with Star Trek. He can be reached via e-mail at john-r@vb-online.com]

Click here to go to the September '95 Article Index