smoores.dev

Over the past five years or so, I've slowly been building up my homelab. What was once just a pihole and a Plex server has evolved into a fairly substantial network of over 40 different services and websites, all run out of a few servers on my home network. It's been one of the best learning opportunities I've ever had, covering Docker, CSP, reverse proxies, DNS, and too much more to list here. Overall, I would highly recommend it.

One crucial piece of network administration has always evaded me, though. Past the routers, VPNs, websites and Docker images, always just out reach, sits the big one: email. I have never been able to wrap my head around even the simplest pieces of email networking. Somehow, this fundamental part of the modern internet has always just been a littlebit too complex, just a little bit too finicky.

Enough Is Enough

This morning when I sat down at my computer, for reasons I can't fully explain, I decided that I'd had it. Today was the day. I was going to get my email properly set up. It was getting ridiculous; I'm a web developer! I even built a simple SMTP server in college for an assignment. I understand the basics, why couldn't I put them into practice?

It's at this point where it might make sense to step back and explain the specific problem I was trying to solve. I have no interest in hosting my own email server (well, not no interest, but I'm not quite there yet), but I do have a Nextcloud instance. Nextcloud is a self hosted Google Drive alternative, and I have it running in a Docker container on my Unraid server. Like any good cloud file storage service, Nextcloud lets you share your files with other users on your instance, or send out share links by email. All you need to do is configure your Nextcloud instance to point to an SMTP server, like the one you use for your normal hosted email, and you're all set.

This is, unfortunately for me, where it gets tricky. I use ProtonMail as my primary email provider. ProtonMail provides an end-to-end encrypted email service, which means that your emails are always encrypted on your end before they're even sent to ProtonMail's servers. They don't actually expose their SMTP service to their customers directly, because then dumb "clients" like Nextcloud, that don't know anything about encrypted email, would just send them emails in plaintext! To get around this problem, ProtonMail provides a piece of software called ProtonMail Bridge. It runs on your computer and exposes a local SMTP server that receives emails from your mail client (like the Apple Mail app or Mozilla Thunderbird, or in my case, Nextcloud) and encrypts them before sending them to ProtonMail's servers. Until recently, ProtonMail Bridge was in beta for Linux, but it was finally released (in a much more polished state) to all of their customers about a month ago.

And So It Begins

Great, I thought to myself, as the sunrise peeked in through the blinds. All I need to do is get the Bridge set up on my Unraid server and point Nextcloud at it and I'll be ready to roll! Well, I can't set it up on my Unraid server, or at least not directly. Unraid is a fork of Slackware, and the OS runs entirely in memory, so if I want to install Linux software that persists between reboots I need to use a virtual machine. Not a problem, I thought, feeling the type of confidence you only feel about 15 minutes into a new project. I already set up a Debian VM to run youtube-dl, I'll just use that.

I logged in to my VM, downloaded the package installer, and with a sudo apt install ./protonmail-bridge.deb, I was off to the races. Sort of. As I mentioned, for a while ProtonMail Bridge for Linux was in a beta program, which I participated in, so at this point I was fairly well practiced in getting it set up correctly on a headless server. This is something it wasn't quited designed for initially, though the latest release does seem to have some better support for it. So I also installed GNU Pass, because the Bridge needs a keychain to store some tokens in, and I generated a new GPG key to initialize the protonmail-bridge folder in pass. As I said, this isn't the first time I've done this, so it only took 4 or 5 tries this time and I managed to get it working just before the headache set in.

So far, you might think, things seem to be going smoothly. Far too smoothly, in fact, to justify the word count of this post. Where's the drama, the intrigue? Well, my sadistic friend, this is where it begins. ProtonMail Bridge, probably for security reasons, binds to the host 127.0.0.1 when it starts. It's actually hard-coded to do so, it can't be configured to find to anything else, which means that it can't be reached from over the network, only from other services running on the same computer. This wouldn't be an issue (remember, technically Bridge and Nextcloud are running on the same computer), except that Nextcloud is running in a Docker container and Bridge is running in a VM, so they think that they're on different machines, which is unfortunately all that matters.

Uphill From Here

This is about the moment that my palms start sweating, as I realize that I might need to take off work for the next week to pore over the docs for sendmail and learn the entirety of email administration just to blindly forward some mail to ProtonMail. But I manage to keep it together long enough to stumble upon E-mailRelay, an email proxy server. This is exactly what I need! It has great documentation and looks like it's kept fairly up-to-date. I frantically downloaded and installed it, feeling very, very greatful for the author of this extremely well documented default config file as I flipped some switches and turned on the service withsudo service emailrelay start. After some brief difficulties getting TLS working between Nextcloud and E-mailRelay (followed by turning it off, it's staying on my LAN anyway!), I was ready to test it out. Heart racing, I clicked the "Send Test" button in my Nextcloud admin panel and I see the successful logs in my terminal and...

emailrelay: info: smtp connection from 192.168.1.73:35962
emailrelay: info: smtp connection closed: smtp protocol done: 192.168.1.73:35962
emailrelay: info: smtp connection to 127.0.0.1:1025
emailrelay: warning: client protocol: unexpected response [Was expecting MAIL arg syntax of FROM:<address>]
emailrelay: info: failing file: "emailrelay.13631.1589033674.3.envelope.busy" -> "emailrelay.13631.1589033674.3.env
emailrelay: error: forwarding: smtp error: unexpected response: Was expecting MAIL arg syntax of FROM:<address>

Wait a minute. What's going on at the end there? That's an error message. And it looks like an error with the syntax that E-mailRelay is using to talk to ProtonMail Bridge? That seems like a Big Deal™, like if nginx was messing up HTTP. I needed to dig deeper and try to understand what was happening, so I installed tcpdump to inspect the communications between the two services. Since the communication was taking place within the VM, I told tcpdump to watch the "loopback" interface and log any packets as ASCII:

$ sudo tcpdump -i lo -A
Bridge > 220 127.0.0.1 ESMTP Service Ready
EmailRelay > EHLO debian.r710
Bridge > 250-Hello debian.r710
Bridge > 250 AUTH PLAIN LOGIN
EmailRelay > AUTH PLAIN [auth token redacted]
Bridge > 235 Authentication succeeded
EmailRelay > MAIL FROM:<email@example.com> BODY=7BIT AUTH=<email@example.com>
Bridge > 501 Was expecting MAIL arg syntax of FROM:<address>

The great thing about SMTP (Simple Mail Transfer Protocol) is that it's a plaintext protocol that reads like conversation. It's clear here that for some reason Bridge doesn't consider the MAIL syntax to be correct. My first question is, of course, is it correct? Time to look through some RFCs.

RFC 5321lays out the specification for the Simple Mail Transfer Protocol, and section 3.3 describes the syntax for the MAIL command.

MAIL FROM:<reverse-path> [SP <mail-parameters> ] <CRLF>

Good news, I guess; the command looks fine! The MAIL command can specify AUTH and BODY parameters after the FROM, so it seems like the issue is on the Bridge side, not E-mailRelay. Time to dig deeper!

Luckily, ProtonMail Bridge is open source, along with the rest of ProtonMail's client software. Unluckily it's written in Go, which is not a language I'm very familiar with. It's at this point where I started making some mistakes. I saw that ProtonMail Bridge depends on go-smtp, an alternative to net/smtp that provides a server implementation and is actively maintained, so I pulled down the code and started reading. And sure enough, there was my error message!

if len(arg) < 6 || strings.ToUpper(arg[0:5]) != "FROM:" { c.WriteResponse(501, EnhancedCode{5, 5, 2}, "Was expecting MAIL arg syntax of FROM:<address>")return} fromArgs := strings.Split(strings.Trim(arg[5:], " "), " ")if c.server.Strict {if !strings.HasPrefix(fromArgs[0], "<") || !strings.HasSuffix(fromArgs[0], ">") { c.WriteResponse(501, EnhancedCode{5, 5, 2}, "Was expecting MAIL arg syntax of FROM:<address>")return} } from := fromArgs[0]if from == " { c.WriteResponse(501, EnhancedCode{5, 5, 2}, "Was expecting MAIL arg syntax of FROM:<address>")return}

In my excitement, I slightly misread this second condition. I skimmed right over the part where arg is split by spaces, so when I saw!strings.HasSuffix(fromArgs[0], ">") I thought I'd found it. Clearly, this library wasn't expecting any of the optional parameters after the FROM arg; it was expecting the last character in the command to be a >. I didn't own this library, or feel very confident about my ability to muddle with the ProtonMail Bridge source code without breaking anything, but I didhave the E-mailRelay source code, which seemed a lot safer to mess around with.

E-mailRelay is written in C++, a language I'm only just slightly more familiar than Go, but luckily SMTP clients are pretty straightforward. It didn't take long to find where the MAIL command was being constructed.

void GSmtp::ClientProtocol::sendMailCore()
{
  std::string mail_from_tail = message()->from() ;
  mail_from_tail.append( 1U , '>' ) ;
  if( m_server_has_8bitmime && message()->eightBit() != -1 )
  {
    mail_from_tail.append( message()->eightBit() ? " BODY=8BITMIME" : " BODY=7BIT" ) ; }
  if( m_authenticated_with_server && message()->fromAuthOut().empty() && !m_sasl->id().empty() )
  {
    // default policy is to use the session authentication id, although
    // this is not strictly conforming with RFC-2554
    mail_from_tail.append( " AUTH=" ) ;
    mail_from_tail.append( G::Xtext::encode(m_sasl->id()) ) ;
  }
  else if( m_authenticated_with_server && G::Xtext::valid(message()->fromAuthOut()) )
  {
    mail_from_tail.append( " AUTH=" ) ;
    mail_from_tail.append( message()->fromAuthOut() ) ;
  }
  else if( m_authenticated_with_server )
  {
    mail_from_tail.append( " AUTH=<>" ) ;
  }
  send( "MAIL FROM:<" , mail_from_tail ) ;
}

I had a hunch that it would be fine to just drop the optional parameters entirely, so that was the first step. I cut out all of the BODY and AUTH logic from the sendMailCore function until all that remained was this:

void GSmtp::ClientProtocol::sendMailCore()
{
  std::string mail_from_tail = message()->from() ;
  mail_from_tail.append( 1U , '>' ) ;
  send( "MAIL FROM:<" , mail_from_tail ) ;
}

The E-mailRelay project, which I'm becoming more impressed with by the minute at this point, has support for installing locally as a .deb package, so testing this out was as simple as ./configure && make deb && sudo apt install ./emailrelay_2.1.deb. And lo and behold, it worked! Simply dropping those parameters from the MAIL command allowed the Bridge to parse the command correctly, and didn't seem to cause any other issues with mail delivery.

Denoument

At this point my system is working, though it doesn't feel great to be relying on a fork of E-mailRelay, especially since my tweak shouldn't have been necessary in the first place! Ideally, ProtonMail Bridge could be updated to properly handle these parameters. And I wasn't alone here; though I might be the only person with this specific set of requirements, during my saga I stumbled upon a reddit post from three months prior where someone was experiencing the exact same issue with KMail, KDE's email client.

My first stop was go-smtp, because I was still under the (slightly misguided) impression that it was being used as the SMTP server interface in ProtonMail Bridge. I opened an issue explaining what I was experiencing. Within minutes, a contributor responded and started digging. It turns out that I hadn't quite gotten it right; adding an AUTH parameter did cause the server to incorrectly throw an error, but it was a different error than the one I was seeing! And while ProtonMail Bridge was using go-smtp, it turns out it was using a very stale fork from 2018.

The contributor had a fix for the current issue ready and merged within an hour, which was outstanding. Unfortunately, this fix would never make it to ProtonMail Bridge if they didn't at least update their fork!Time for another issue, this one in the Bridge repo, and armed with a lot more information. With any luck, fixing this will be as simple as pulling in the latest go-smtp!

This rabbit hole consumed just about my entire day, but on the whole it actuallyenergized me. It feels good to be able to contribute to these projects like ProtonMail Bridge and go-smtp, even if it's just by identifying and flagging bugs in a useful way. And it feels good to have enough working knowledge to patch an issue locally so that it doesn't break my own system! Everything about web development's foundations, from open protocols like SMTP to the RFC process supports this kind of contribution, and I'm finally starting to be able to take part.

Epilogue

May 11, 2020

With some fresh eyes, I took another look at ProtonMail Bridge's go.mod file this morning. I noticed that they were specifying a commit to depend on, and after some digging learned that that commit was actually much older than I thought it was! It wasn't using the code I shared above at all, instead it was using a regex to validate the MAIL command:

re := regexp.MustCompile("(?i)^FROM:\\s*<((?:\\\\>|[^>])+|\"[^\"]+\"@[^>]+)>( [\\w= ]+)?$")
m := re.FindStringSubmatch(arg)

This regex is so close to working, but it exclusively expects word characters in the optional parameters (because of the \w), which means that it breaks when the client sends an AUTH parameter like AUTH=<email@example.com>.

The very, very good news is that ProtonMail responded to that issue, and they're planning on updating to the latest go-smtp!