Google, Microsoft, Yahoo and others publish new email security standard

Devised by engineers from Google, Microsoft, Yahoo, Comcast, LinkedIn and 1&1 Mail & Media Development & Technology, the SMTP Strict Transport Security is a new mechanism that allows email providers to define policies and rules for establishing encrypted email communications.

The new mechanism is defined in a draft that was published late last week for consideration as an Internet Engineering Task Force (IETF) standard.

The Simple Mail Transfer Protocol (SMTP), which is used to transfer email messages between email clients and servers, as well as from one provider to another, dates back to 1982 and was not built with any encryption option.

For this reason, in 2002, an extension called STARTTLS was added to the protocol as a way to include TLS (Transport Layer Security) with SMTP connections. Unfortunately, over the following decade, the extension was not widely adopted, and email traffic exchanged between servers remained largely unencrypted.

That changed after 2013, when former U.S. National Security Agency contractor Edward Snowden leaked secret documents that revealed widespread surveillance of Internet communications by intelligence agencies from the U.S., U.K., and other countries.

In May 2014, Facebook, which sends billions of notification emails to users every day, ran a test and found that 58 percent of those emails passed through a connection encrypted with STARTTLS. By August that same year, the rate jumped to 95 percent.

There is one problem, though: Unlike HTTPS (HTTP Secure), STARTTLS allows for what is known as opportunistic encryption. It does not validate the digital certificates presented by email servers, under the assumption that even if a server's identity cannot be verified, encrypting the traffic is still better than nothing.

This means that STARTTLS connections are vulnerable to man-in-the-middle attacks, where a hacker in a position to intercept the traffic could present the email sender with any certificate, even a self-signed one, and it will be accepted, allowing for the traffic to be decrypted. Furthermore, STARTTLS connections are vulnerable to so-called encryption downgrade attacks, where the encryption is simply removed.

The newly proposed SMTP Strict Transport Security (SMTP STS) addresses both of those issues. It gives email providers the means to inform connecting clients that TLS is available and should be used. It also tells them how the presented certificate should be validated and what should happen if a TLS connection cannot be safely negotiated.

These SMTP STS policies are defined through special DNS records added to the email server's domain name. The protocol provides mechanisms for clients to automatically validate these policies and to report back on any failures.

Servers can also tell clients to cache their SMTP STS policies for a specific amount of time, in order to prevent man-in-the-middle attackers from serving fraudulent policies when they attempt to connect.

The proposed protocol is similar to the HTTP Strict Transport Security (HSTS), which is meant to prevent HTTPS downgrade attacks by caching a domain's HTTPS policy locally in the browser. It does, however, assume that the first connection from a particular client to the server was performed without being intercepted; otherwise, a fraudulent policy might have been cached.

According to Google's latest data, 83 percent of email messages sent by Gmail users to other email providers from around the world are encrypted, but only 69 percent of incoming emails from other providers are received over an encrypted channel.

There are also large discrepancies in email encryption between regions of the world, with email providers in Asia and Africa faring much worse than providers in Europe and the U.S.