There are essentially two types of protocol used in the e-mail process: transport protocols, and storage and retrieval protocols. We will look at these protocols and the overall mail process in some detail, before moving onto programming e-mail in the next section.

The Mail Process

The sender of an e-mail message uses a program to create and send mail; this program is known as the Mail User Agent (MUA). Once created, the message must be moved to the recipient's mail server over a transport medium - the Internet or a company's intranet for example - using a Mail Transfer Agent (MTA). The message is then delivered to the recipient using a Mail Delivery Agent (MDA) and stored until the recipient chooses to read it using his or her MUA.

So, the most important mechanisms in the mail transport system are:

• Mail User Agent (MUA) - A program used to create and receive mail messages
• Mail Transfer Agent (MTA) -The means by which mail messages are transferred between machines over the Internet
• Mail Delivery Agent (MDA) - The mechanism that delivers the mail message to the recipient's mailbox when mail is delivered via an MTA to the mail server

The Protocols of Internet Mail

To understand how to code applications using e-mail functionality, it is unfortunately necessary to understand the large number of acronyms that come with the territory! Some of these acronyms are associated with e-mail protocols, which have evolved into sophisticated messaging tools. Thus, we will now take a look at the following questions:

• What are RFC 822 and MIME?
• What are S/MIME and OpenPGP?
• What is SMTP/E-SMTP, how does it work?
• What are POP and IMAP?
• How does LDAP fit into mail systems?
• How does ACAP fit into mail systems?

Programming WAP applications with e-mail capabilities will involve some or all of these protocols.

Mail format and transport protocols

In this section we will look at the mail protocols associated with formatting and transportation of messages. This will in effect answer the first three questions asked above.

On the Internet, Mail User Agents (MUAs) typically send messages in a MIME encoded format to Mail Transfer Agents (MTAs), which transport messages using SMTP/ESMTP protocols.

RFC 822

In the past, e-mails were sent in a standard format, specified in RFC (Request For Comment) 822, entitled 'Standard for the format of ARPA Internet text messages'. This encoded the mail as plain text in a US-ASCII 7-bit character set format with no multi-part structure to the message body. The need for the inclusion of other language's character sets, and multi-media attachments led to a need for a more complex message structure, which was solved by the creation of MIME.

MIME

MIME (Multipurpose Internet Mail Extensions) defines the necessary message structure (see RFC 2045-2049) needed to work with different 8-bit character sets and multi-part messages. MIME was constructed as an extension to RFC 822, allowing older MUAs to continue to work, ignoring the new features, formats and extensions. MIME specifies a Content-Type header that can be used to specify a character set or media type for the e-mail message, for example:

   Content-Type: text/plain; charset=us-ascii

This indicates that the message is in plain text using the traditional US-ASCII character set. However, it also has the ability to specify more exciting types, such as images and other binary attachments.

In a typical e-mail message, you may see several attachments indicated by MIME type boundaries. For example the following e-mail header snippet indicates the email it came from contains a simple text section, followed by a Microsoft Word document (Content-Type: application/msword) as an attachment:

   ------_=_NextPart_000_01BFA0AF.11763546
 Content-Type: text/plain;
    charset="iso-8859-1"

 ------_=_NextPart_000_01BFA0AF.11763546

 Content-Type: application/msword;
    name="Speaker FAQ.doc"
 Content-Transfer-Encoding: base64
 Content-Disposition: attachment;
    filename="Speaker FAQ.doc"

 ------_=_NextPart_000_01BFA0AF.11763546--

It is the responsibility of the receiving MUA to correctly interpret and display the message based on the specified type(s).

End-to-End Security

Security services can be added to each step of the communication path, for example by using Transaction Layer Security (TLS), also known as Secure Socket Layer (SSL). Alternatively, the security can be 'wrapped around' the data being transmitted, making it independent from the transport mechanism. This second type of security is known as 'end-to-end' security. S/MIME (Secure MIME) and OpenPGP (Open Pretty Good Privacy) are two competing protocols that provide 'end-to-end' security for e-mail messages.

S/MIME and OpenPGP provide all the functionality of MIME but, in addition, allow you to sign e-mail, and send it in a secure 'envelope'. LDAP (a directory service which is discussed in the next chapter) and X.509 certificates can also be used to secure Internet mail.

Work is currently being done by the IMC (Internet Mail Consortium), and the IETF (Internet Engineering Task Force), to resolve S/MIME and OpenPGP, which represent two incompatible standards for mail encryption and security, into a single standard.

S/MIME

S/MIME was originally developed by RSA Data Security Inc. and is based on the Public-Key Cryptography Standards (PKCS) #7 Cryptographic Message Syntax (RFC 2315), and the X.509v3 certificate format. S/MIME creates a message digest, which is encrypted to ensure the message cannot be tampered with in transit. In order to 'seal the message envelope', S/MIME encrypts all the contents using a triple-DES key algorithm.

OpenPGP

OpenPGP, which has superseded the PGP/MIME format, is based on the Pretty Good Privacy (PGP) data encryption standard.

SMTP

A transfer protocol is the language spoken by Mail Transfer Agents (MTAs). There are many such protocols in existence, such as X.400, UUCP (Unix to Unix Copy), and SMTP (Simple Mail Transfer Protocol). SMTP is by far the most commonly used and is the standard message interchange protocol on the Internet.

SMTP uses an envelope and body metaphor to structure a mail transfer. The envelope is used to transfer messages and contains information about the message sender and the destination address. The originating address is used to notify the sender of the message when a delivery failure occurs. The SMTP body contains the entire message including the body and header information.

E-SMTP

Extended Simple Mail Transfer Protocol (E-SMTP), which is described in RFC 1869, is an extension of SMTP to allow for an 8-bit character set. The 7-bit US-ASCII character set, which SMTP used, needed to be extended to allow for other European and Asian characters and to cope with multi-media message bodies. This is very much in line with the MIME extensions to RFC 822.

Storage and Retrieval protocols

When a user retrieves mail from his Internet Service Provider (ISP) he will almost certainly be using POP3 or IMAP4, as the retrieval protocol.

POP

The Post Office Protocol (POP3) would notbe necessary if users had a permanent connection to the Internet and received SMTP messages directly. Historically, users would dial into such a permanently connected machine to read their e-mail remotely; this mail server would be constantly connected to the transport medium of the Internet, allowing it to receive mail continuously.

With the growth in personal computing, there was increased demand for e-mail to be stored locally, with only sporadic connections to the Internet. This led to the development of POP3 as a protocol
to retrieve mail from a remote mailbox, via the user's MUA, allowing the mail to be read and stored locally.

IMAP

The most commonly used mail retrieval mechanism is still the Internet's POP3. However, the Interactive Mail Access Protocol (IMAP4) is gaining ground fast.

While POP3 stores all messages in a single message queue, IMAP4 allows for a more flexible mail storage metaphor, using a hierarchical folder concept in which to store messages, and providing a richer set of mail manipulation functions.

IMAP was designed with the following goals in mind:

• To provide the ability to store mail in folders besides your own inbox
• To provide a richer set of functionality for manipulating mail folders
• To provide better access to a mailbox in online and disconnected access modes than the simple offline access offered by POP3
• To facilitate access to a user's email from more than one client computer (for example from a mobile device and an office PC)
• To be fully compatible with the open standards of Internet messaging such as RFC822, MIME, and E-SMTP

As people access their mailboxes from a wider and wider variety of different locations and devices, IMAP4 will come into its own as a more comprehensive mail storage protocol than the traditional POP3. IMAP4 will continue to grow in popularity, replacing POP3, as the requirement for more flexible and powerful e-mail storage increases.

However, as POP3 is still the dominant protocol in use on the Internet, and as it is slightly easier to program and conceptualize, the examples using mail retrieval in this chapter will use POP3.

E-Mail's Relationship with LDAP

Ever since the development of X.400 and X.500, e-mail services and directory services have been functionally linked. Lightweight Directory Access Protocol (LDAP), which is the subject of the next chapter, is by far the most commonly deployed directory service for storing corporate address book information, and can be used to deliver address book functionality to MUAs. This is particularly useful to mobile users: there is little benefit in storing globally-administered information locally when it can be efficiently accessed from a single central source, thereby eliminating duplication and the need for synchronization.

E-Mail's Relationship with ACAP

The Application Configuration Access Protocol (ACAP) was conceived at Carnegie Mellon University as part of an ongoing project to enhance the university's e-mail and message exchange systems, known as Project Cyrus. ACAP has been proposed as an Internet standard as defined in RFC 2244.

ACAP covers a lot of the same functionality as LDAP in that it aims to store attributes (name/value pairs) relating to a user on a centrally accessible server. ACAP's focus is to allow highly mobile users the ability to retain configuration information and preferences specific to an individual user - for example, implementing a personal address book - and has been used by some MUAs for this purpose. ACAP client software is also designed to work offline, unlike LDAP.

Work has been done to closely integrate ACAP and the IMAP mail protocol, and if these services achieve a wide market penetration, they may provide substantial benefits to mobile users. However, ACAP suffers from not being as broad in its approach as LDAP, which is ahead in terms of both the maturity of the protocol, and its market share.

A Typical E-Mail System

To summarize, it might be helpful to look at some diagrams of a typical mail system using the Internet protocols. From a user's point of view e-mail is sent via SMTP, collected from their mailbox using POP3 or IMAP, and any address book information is searched for using LDAP (or ACAP):

In terms of the flow of a message (either a RFC822, or a MIME, formatted message) from sender to recipient, the transmission path looks like this:

You should now understand how a mail system is structured. We have looked at how a Mail User Agent talks to a Mail Transfer Agent to send e-mail across a mail backbone, how mail is retrieved from a store and how LDAP is accessed for address book information.

Our next task is to look at how we can incorporate this mail functionality into our mobile applications, using WAP and e-mail to deliver real business value.

Programming E-Mail

There are a number of messaging APIs (Application Programming Interfaces) on the market that enable developers to write simple and 'generic' mail functionality into their programs. Before we look at some specific code examples, and consider how to construct applications that make use of messaging functionality, we will now look at the major API sets, which are:

• Common Mail Calls
• Vendor Independent Messaging
• CDO and CDONTS
• JavaMail

Common Mail Calls

Common Mail Calls (CMC) is the X.400 standard mail API set developed by the X.400 API Association (XAPIA), and as such, has a limited application to this chapter since we are dealing with the Internet mail standards - SMTP, POP3, etc - rather than X.400.

Vendor Independent Messaging

Developed by Lotus, Vendor Independent Messaging (VIM) provides a simple set of API calls that provide platform-independent access support for mail systems.

Microsoft's Collaboration Data Objects (CDO)

CDO is an extremely simple-to-use API that is based on COM objects. What this means in practice is that it's the API you will want to use if you are coding in Visual Basic, Visual C++, or scripting languages on Microsoft platforms - which will be the case if you are using ASP to generate WML pages dynamically - we'll look an example of how to do this later in the chapter. CDO comes in two variants, CDO and CDONTS (CDO for NTServer).

CDO is designed to run against Microsoft's Exchange Server - a sophisticated mail server that can send and receive mail and organize all users' mail accounts for a large organization. Exchange Server's functionality is considerable and is accessible practically in its entirety from CDO. That means that although CDO is still relatively simple to use, it is more complex than you might want for very basic send/receive email functionality. For this reason Microsoft developed CDONTS as an alternative. CDONTS can run either against Exchange Server or against the simple SMTP service that is shipped with Internet Information Server (IIS). CDONTS can't do much beyond sending and receiving e-mails, but that means it is incredibly easy to use - as we'll see later on when we look at an example using CDONTS to programmatically send an email with just three lines of VBScript code.

Using CDO, complex messaging functionality can be pulled together with other related COM based programming models such as ActiveX Data Objects (ADO) and the Active Directory Service Interface (ADSI). This linkage will provide mechanisms to manipulate mail, and easily extend CDO applications, to include other areas of personal information management functionality, such as contact and calendar management.

JavaMail

Released by Sun in August 1998, JavaMail aims to revolutionize access to mail systems in the way that Java Database Connectivity (JDBC) revolutionized access to databases. We will be dealing with this at length in the next section. The advantages of using JavaMail include:

• It is available on the majority of current operating systems
• It offers an e-mail API that is both flexible and easy to use
• It offers excellent networking capabilities as standard

In the public imagination, Java and the Internet are inextricably linked, and to the programmer Java continues to offer the simplest and most elegant language with which to implement network-centered applications. The wealth of networking APIs and server-side solutions offered by Java makes it an obvious choice for implementing the distributed backend systems needed for our WAP applications.

We can make use of Java on the server-side to effectively scale and efficiently partition our WAP applications. With JavaMail, Sun has provided an essential API that will enable us to give WAP advanced mail functionality, adding value to our m-commerce solutions.