The Secrets of the Ancients

That Nathan Fritz. He’s a smart guy, and always been very nice to me, at least, if we discount that incident. And over on the &Yet blog, he’s posted about sending hints rather than data. And when I read it, my heart sank.

Now, I should stress, my heart didn’t sink because Fritzy is in any way wrong – he’s not – it sank because not once in the entire article did he use the P-word: “Protocol”. So I’m going to, and as I do so, I’m going to reveal – drum roll please – the Secrets Of The Ancients. These are things which Fritzy knows very well indeed, but that’s because Fritzy – like me – is a Protocol Designer.

My story begins in a computer lab in the far off and fabled land of Stanford, back in the hazy years known only as the ’80’s. In it, a young and enthusiastic new Systems Programmer was dabbling with this new-fangled thing called the Internet. His name was Mark Crispin, and he was about to have an epiphany.

As well as writing the truly awesome RFC 748, the very first April 1st RFC (a tradition that continues to this day, and spread to the XSF as well), he started using the the Lisp workstations to read his mail. This would launch him on a trajectory that would define his life’s work.

Up until this point, people read their email generally by using a terminal directly attached to a particular mainframe, or else using a workstation to emulate one over the Internet (either via TELNET or SUPDUP – both of which Mark worked on). But Mark observed that users generally wanted to read their mail on their workstation – whichever workstation that happened to be that day. And so he helped invent a concept where mail was delivered to a central mail server and then workstations could simply access the mail there, and in doing so, he created the Interim Mail Access Protocol.

In many respects, you can consider IMAP to be the very first cloud service – mail was held and managed for the user on some specialist mail server somewhere on the network.

In 1988, when Mark published IMAP2 (Interactive Mail Access Protocol) in RFC 1064 as an “experimental specification for the Internet community”, another protocol was also in existence in the same space – POP2. POP2 operated in a simple manner – you asked for data, and you got it. There was little complexity. However, it couldn’t notify you that there was new mail to fetch – you just asked every now and then to see if there was any. And if there was, you fetched it all.

“In defining a […] protocol, it is important to keep in mind that the client and server form a macrosystem, in which it should be possible to exploit the strong points of both while compensating for each other’s weaknesses.”

Mark avoided any requirements on the client which were already managed by the server, so clients never held any long-term state, and didn’t need to be registered with the server – you just needed your username and password, and you could connect to the server and start reading your mail.

IMAP2 – like its later variants that are in use today – also didn’t have a concept of commands which fetch data. This may come as a surprise to many people who’ve implemented IMAP, but as RFC 1064 says:

“The actual data is transmitted via the unsolicited [push] data mechanism (that is, FETCH should be viewed as poking the server to include the desired data along with any other data it wishes to transmit).”

The server could push whatever it felt like, but typically the client would ask for the “envelope” – the from, to, subject, and so on – rather than the complete message, therefore saving the effort for pulling the actual message until the user actually cared.

Mark then moved on to be a key player in the development on MIME, the mechanism which allows attachments and complex messages on the Internet – during which time he argued strenuously that the “MIME-Version” header was a bad idea; version numbers, he claimed, had no place in protocol design – IMAP, despite having version numbers, actually works via capability strings – years later web developers would realise version numbers were an anti-pattern.

By IMAP4 (the Internet Message Access Protocol) – itself 20 years old now – servers were allowed to push data even when the client wasn’t running a command, and clients could – if they wanted – maintain a cache of message data they could access offline, and resynchronise when they reconnected to the server. They could access partial messages – just the HTML, this attachment but not that one.  At the same time, Javascript was still a year away from inception.

Now, almost every mail provider in the world offers IMAP4rev1, and clients are available on every platform imaginable.

The protocol operates on the following basic principles:

  1. The server maintains state on a per-object basis, not on a per client basis. Individual clients should never have to be tracked by the server outside of a session.
  2. Active clients should always be kept informed about what data exists. This is done by informing them at the beginning of a session, and pushing updates continuously through the session.
  3. Clients never need to track the actual data – just its presence. In IMAP, the actual message bodies aren’t sent to the client except on request; just the existence of the message is all that’s sent – sufficient metadata to allow clients to present the item to the user, or simply present the item’s existence, is enough.
  4. All data is, conceptually, pushed. In order to explicitly request data, clients request that it be pushed.
  5. Wherever possible, make data within the model static. In IMAP, messages are mostly static data, with dynamic state reduced to read-state and other flags. Complex changes are much harder to represent; it’s easier to consider either the entire item replaced, or (as IMAP does) the item deleted and an entirely new item added.
  6. Command-Response is bad. Clients should be allowed to send Command1-Command2-Command3, and server should be able to respond Response2-Response1-ResponseX-Response3. This is made much easier by (4).

And now, the Web has finally got to the point where Web Applications can also take advantage of the protocol architectures that Mark created decades ago. WebSockets and HTTP/2 allow for server push (which Mark had explicitly documented by 1994), LocalStorage and ServiceWorkers allow for client-side stored data and code, and even disconnected mode operations. Ancient secrets, like IMAP, have suddenly become relevant again.

Finally, after setting networked computing back almost 30 years, the web is able to start catching up again. Mark Crispin died at the end of 2012, aged 56. His legacy is only just getting started.