smoores.dev

The internet as a system is inherently public. The primary protocols, like HTTP for website traffic and SMTP for email traffic, are plaintext protocols, and the underlying Internet Protocol (IP) includes fairly stable addresses (IP addresses) in every request, which means that the computers that serve content, like web servers and DNS servers, are constantly being informed about who, specifically, is making which request.

This isn't bad, per se. In some ways it's inescapable; if you want to retrieve the Wikipedia page for, say, the Internet Protocol, at some point you have to tell the server that owns that page where to send it. And the stability of IP addresses is good, too. It's like including a return address on a physical piece of mail. The stability allows the network to learn the fastest routes from one computer to another, and it allows various parts of the network to cache certain information about other parts.

But the content servers aren't the only ones that can see who is making which request. Along the way from your personal computer to Wikipedia's servers, there are dozens of other "hops" involving computers owned by third parties. Because of the transparent nature of these protocols, any of these third parties, from your landlord's router to your cable company, is able to see, and even intercept and manipulate, the traffic that passes through it.

Luckily, most web traffic nowadays uses the HTTPS protocol. The S stands for Secure (no, really), and this security extension to the HTTP protocol has not one, but two primary security measures. The first is encryption; using Transport Layer Security (TLS), HTTPS ensures that any computer "sniffing" network traffic will only see unintelligible gibberish, and only the two computers at either end get the actual contents. This means that even if your request included sensitive information like login credentials or a social security number, only the intended recipient would actually be able to see that information.

The second security measure is "authentication", which is managed by the Certificate Authority system. Each website can register with a trusted Certificate Authority, and in turn receive a unique TLS certificate from that authority. Because the certificates are unique, any client receiving information from a given website can check the certificate against the registry that delivered it and know with certainty that they are in fact talking to the real Wikipedia.

So, What's the Issue?

HTTPS was a massive step forward for security on the internet, and made a fairly large improvement to privacy, as well. Unfortunately, it was a fundamentally security-motivated change, and so there are still some privacy-shaped holes.

Most of the remaining privacy issues around web protocols involve DNS, the Domain Name System. DNS is the address registry that computers use to translate human-readable domain names, like smoores.dev, into computer-readable IP addresses, like 24.59.60.32. Queries to DNS servers are still sent in plain text, which means that even if your network doesn't use your ISP's DNS server (they almost always do by default), your ISP and other neutral/bad actors are likely still inspecting your DNS traffic.

To be clear, this is still a massive improvement over unencrypted HTTP. Only the domain name itself is visible; all of the data, even the path of the URL, is encrypted and unsniffable. But there are still places in the world where even looking at certain websites can be extremely dangerous, and third-party marketing and data collection agencies can still create shockingly accurate profiles based on nothing other than how often users visit which websites. HTTPS is better, but it's not enough.

Closing the Gap

It seems like the right step forward is to somehow encrypt the DNS query itself, and in fact, that's exactly the direction we're headed. There are a couple of competing standards, the two primary ones being DoH (DNS over HTTPS) and DoT (DNS over TLS). DoT involves encrypting DNS queries with TLS, the same encryption layer as HTTPS, whereas DoH actually entirely uses the HTTPS protocol for DNS queries.

Turning this on on my home network was about as straightforward as anything could be. My router is running Asuswrt-Merlin, which has built-in support for DoT, and includes presets for Cloudflare's DoT servers. My router's DHCP server points all of the computers on the network to my pihole, which forwards any unblocked queries back to the router, and onward to Cloudflare, fully encrypted!

Encrypted DNS queries solve a huge privacy problem: they allow us to privately retrieve the IP address of websites that we want to access. Just like with the advent of HTTPS, third-parties like ISPs will only see random bits when attempting to sniff DNS queries made over DoH or DoT.

Unfortunately, there's another gap in the system, as well. There are actually two different places that computers using modern day HTTPS need to specify the requested domain name in plaintext: the DNS query, which we just discussed, and the Server Name Indication, or SNI. SNI is an extension to the TLS specification (the system used to encrypt HTTPS traffic) that enables servers to host multiple TLS-enabled websites on the same IP address. For example, this server hosts smoores.dev, ink.talks.smoores.dev, and resume.smoores.dev, along with a few others. Each of these domain names has a different TLS certificate associated with it, and it's important that the server knows which one to use for which requests, so that it can support the authentication component of HTTPS.

The trouble is, because the domain name determines the TLS certificate to use, it needs to be sent unencrypted, so that it can be read and used by the server before the encryption handshake occurs. You can read a slightly more technical explanation of this process here, if you're interested. This means we're back to square one, where a very determined malicious coffee shop owner can still track the domain names their customers are visiting, even with encrypted DNS!

There's a solution, though, or at least the beginning of one. The reason I linked to an article by Cloudflare above is because Cloudflare is working with Mozilla, creators of Firefox, to prototype a new extension to TLS, Encrypted SNI, or ESNI. ESNI works by allowing servers to publish a public encryption key to a new type of DNS record, which can be fetched during encrypted DNS queries. The client, in this case the Firefox web browser, can use this public key to derive an encryption key that the server also has the ability to derive (using its private key), and thus encrypt the SNI in a way that the server can decrypt, before it's ever send a message to the server! Again, check out the next section of that Cloudflare article for more technical details.

Actually Closing the Gap, Like, for Real Though

Now that we understand what ESNI is, we need to actually set it up to use it. ESNI relies on a suite of underlying technologies: HTTPS (specifically over TLS v1.3), encrypted DNS (either DoH or DoT), and DNSSEC (a protocol for validating that DNS records are legitimate). In most setups, all three of these technologies come for free if you're using Firefox as your web browser. Firefox always uses the latest version of TLS, and its internal DNS client supports DoH and DNSSEC. You can enable all of these features in the advanced settings of the browser, where you'll also find the setting for enabling ESNI.

My setup is a little different, though. I use a pihole to filter web traffic at the DNS level, which means that all DNS traffic on my network hits my pihole first, then my router, then it goes off to Cloudflare. The first two hops are unencrypted; since they're within my home network, there's no concern that a third party could be inspecting them. The last hop, from my router to Cloudflare, is encrypted with DoT.

The problem is, Firefox won't enable ESNI unless you're also using encrypted DNS, and from Firefox's perspective, my DNS server (the pihole), doesn't support encrypted DNS! I essentially needed to trick Firefox into thinking that it was sending DoH traffic to the pihole, knowing that the traffic would actually be encrypted by the time it left my network (via the DoT settings on the router).

The solution was an experimental DoH proxy server built by, of all things, Facebook. doh-proxy is a python server that can terminate TLS on DoH requests and then forward them as plain DNS queries to an upstream DNS resolver. It ships with doh-httpproxy, which is meant to be run behind a reverse proxy. This is perfect for me, because I use Caddy as a reverse proxy for all of my internal and external self-hosted projects. I can start it up with doh-httpproxy --upstream-resolver=localhost --port 9483 on the same server that my pihole lives on, and it'll proxy DoH requests from Firefox to the pihole over standard DNS, leaving Firefox none the wiser. I can manually configure Firefox to use my local DoH server for DNS resolution in the standard Firefox DoH settings, and suddenly, I have ESNI!

Hopefully, encrypted DNS and ESNI will catch on more broadly, and be more widely supported across all browsers and DNS providers. For now, at least when I'm on my desktop or laptop (and visiting a site hosted on Cloudflare), I know that my browsing habits aren't being consumed, saved, and monitized without my consent.

Check out the IETF draft on ESNI for all the juicy technical details about the proposal.