A Modern Message Transfer Agent

At Genius.com, with ever-increasing customer growth and demand, we began looking for a Message Transfer Agent (MTA) that would fit our needs and would integrate quickly with our application. One feature of our application handles large email marketing campaigns on the behalf of our customers and our existing MTA was not providing us the visibility and control we needed. We were not looking for an MTA to  help us create campaigns, but something that would take the mail and pass it to the various recipients, reliably, and efficiently.

We looked at many systems; we looked at open-source, we looked at closed-source and ultimately, after much deliberation, selected MessageSystems.

Let me explain what makes MessageSystems unique for us.

Managing at Scale: Clustering environment

MessageSystems can be setup in a cluster with several active nodes and a manager. The nodes are bound to a range of  Internet Addresses and can seamlessly exchange these addresses between nodes depending on which node is available. The configuration is managed from a subversion repository and nodes download the latest version at startup. This makes it particularly easy to centrally manage the configuration of the system. All the nodes are active: no machine is collecting dust waiting for its big day.

Logs for each node are transferred to the manager and stored by date. While the logs rotate on the nodes, the manager will keep the logs indefinitely. This interests us as these logs are important for extracting historical data and finding the reputation and performance of a set of customers over time. With this information we can talk to our customers and help them to improve their email processes and marketing campaigns.

Sender Reputation Management: Not all IP addresses are equal

What also makes MessageSystems unique is the handling of IP addresses. An IP address can be assigned for a particular class of email, customer, or recipient. This is useful when we want to offer dedicated service to our customers. Most MTAs will check the inbound MTA IP address, and the name the sending MTA is advertising via the HELO command. They will check that the name matches back to the IP address, and they will check that the IP address resolves back to the name. Having the capability to differentiate addresses is important. Most reputation systems are based on IPs. Which IP is doing what. For instance senderscore or senderbase will track the reputation of IPs. Being able to control on which IP a particular email will be sent, allows us to control our reputation based on our customers reputation.

Bounce Processing: How to flexibly handle bounced emails

Most Non Delivery Reports (NDR) are totally opaque for the common mortals.  Most users ignore them, which results in Support requests like:

User: “why couldn’t I send this email?”
Support: ”Did you read the error report?”
User: ”The gibberish email you mean?”.

Each MTA out there has its own unique way to report that an email could not be delivered. It may happen at three different stages:

  • The first stage is when you try to send the email, your MTA is trying to talk to the remote MTA, and cannot find it; either the domain name does not resolve, or the remote MTA is not available.
  • The second stage is when your MTA talks to the remote MTA, and the remote MTA spits a dreadful 500 error code, with a one liner indicating the reason of the reject. In these 2 stages, most MTAs allow you to customize the error message to send back to the user, providing a bit of  sense in what happened.
  • The last stage is when your email is successfully sent to the remote MTA, but the remote MTA decides after all, it does not want to accept it. For instance it has scanned the message and found a virus, or it was only a front to another MTA inside the corporate network. The user is then subjected to the remote MTA NDR templates. The most difficult NDR to interpret are internal NDR generated by MS-Exchange because of the conversion between SMTP and X400, and that’s when Exchange does not send an error in proprietary MS-TNEF format.

Well, if you tell MessageSystems what a bounce or NDR looks like (the bounce address), it is able to catch the NDRs and via heuristics, classify them. You can then report to the user, with your own specific message. Or in our case, simply log all the bounces and their classification. We differentiate between soft and hard bounces automatically. If we had to do this classification ourselves we would have to study all the different mail software out there and continually maintain the classification mappings. MessageSystems does that for us.

Domain Based Throttling: Working with receiving MTAs to be a good citizen

Each receiving MTA has its own particulars, it is important to be able to adapt to the standards of each MTA to which we send emails. To protect against spam, mainly coming from botnets, MTAs will:

  • Check against Realtime Blocking Lists (RBL)
  • Check the history of the sender
  • Check the content of the message

and will adapt depending on the reputation of the sender. The receiving MTA may decide to not accept more than one message per minute, not more than one connection at a time, and/or not more than one recipient per message. Each parameter needs to be controlled. MessageSystems allows us to configure per receiving domain what should be the rate of send so the receiving MTA can process the messages at its own rhythm.

Customization: Sieve ++

What is unique, is the way MessageSystems processes emails. It has its own scripting language based on Sieve called Sieve++. It is a bit like the Sendmail rule based system, but you access all the parameters of an email at various stages of the processing. For instance in sendmail, you have access mainly to the email of the recipient, there you can decide what you want to do with the email.

With MessageSystems you can make decisions at each phase of the:

  • Accept
  • Connect
  • Ehlo
  • mailfrom
  • rcpt to
  • data
  • each_rcpt
  • set_binding

As we are mainly sending emails, we do not worry too much about receiving emails, we just make sure we are not an open relay, accept all messages and pass on the messages to be processed for bounce, or abuse.

For sending, we like to study the message and run a few checks. For instance, we keep a list of emails from people and domains that have requested us to not include them in the marketing campaigns of our customers. In Sieve++ you would do it like this:

# suppression list
require["vctx_conn", "ds_fetch", "ec_test", "ec_reject", "ec_log"];
$bounce = envelope "from";
# suppression list by email
$recipient = envelope "to";
($status) = ds_fetch "db" "select status from suppressionList where email =?" [$recipient];
if ec_test :is "${status}" "1" {
    ec_log "Blocked due to Genius Policy: email";
    ec_reject "Blocked due to Genius Policy: email";
}

Sieve is a scripting language created to offer a simple way to take actions on the emails you receive in your inbox (filtering). Many email clients use methods to move an email from the inbox to a specific folder. This type of filtering does not work if the email client is not running. The mail server should be able to process the messages and place them in the right folders, but with many clients and many servers, what system can you use? Here comes sieve, a simple language that email clients can understand. Email clients send scripts to the email server so it can use them to process emails each time an email is received in a particular inbox.

With MessageSystems, sieve is extended in 2 important and valuable ways:

  • sieve has new commands
  • sieve is used to process emails before they are even delivered to the mailbox.

Let’s take SpamAssassin, a common spam detector, and Postfix, a popular email system. Usually Postfix, in the middle of its message processing, would pass the message to a program like SpamAssassin. SpaAassassin would analyse the message, modify it, and pass it back to postfix to deliver to the mailbox or to reject it. This does not give too many options. If you wanted to do some fancy things like the script above, you would have to ensure you have built a robust script which would not drop messages. With sieve++ linked at each phase of the message processing, you can deliver, discard, reject, modify, redirect, add more recipients to the message, and many other possibilities, without worrying to much about errors or losing emails.

In our case, with the above script, we get the recipient email, query a database to see if this email is in our list, and, if the test is successful, we log the reason and reject the message (this will create a NDR with the reason as stated).

Data Analysis: Log analysis is lacking

The way MessageSystems analyzes logs is not optimal for our processes. It is designed to provide visibility when running MessageSystems as a corporate mail server, but it does not give us information about our customers’ activities over time, or our reputation for a particular sending IP and/or receiving domain.

Instead, we have built tools that pick up the extensive and custom logs and store them in a database in a format which is linkable to the objects in our web application. From these, we built real-time dashboards and overviews summarizing several months of our deliverability and reputation, globally as well as per customer.

Conclusion

MessageSystems has allowed us to effectively separate the email functions from our application transactions and provides us the tools and visibility necessary to efficiently manage and improve our reputation and that of our customers. Building these capabilities into an existing email system is possible, but requires significantly more resources and a longer time between the idea and its deployment.

If you are looking for a high performance, highly redundant and manageable MTA, definitely consider MessageSystems.

  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Twitter
  • Google Bookmarks
  • DZone
  • HackerNews
  • LinkedIn
  • Reddit
  • http://www.messagesystems.com Mike Petonic

    It’s great to see you guys really use our platform to its fullest, and thank you Genius-Engineering-Team for your praise (you guys are really Geniuses!).

    Wait until you see our 3.0 release… Lua scripting. As powerful as Sieve++ is, Lua is a lot more powerful, and more intuitive, to boot. It’s kind of like the difference between programming in Lisp to programming in Perl.

    But in seriousness, it’s great to see our product get used to its fullest extent instead of just looking at a mail delivery engine as a “spam cannon.” Genius is the type of company that is bringing relevance back into email, and helping to make it meaningful again for its recipients. We look to you and your team’s continued feedback as we continue to make our platform more valuable to our power-users.