The blog
AUG19

The Bug That Wouldn't Die

A band tried to remove one email address from their forwarding list and got a generic server error instead. Simple bug, we thought. It was not a simple bug.

Attempt one: read the Cloudflare error, fix what it seemed to say, ship it. Still broken, differently. Attempt two: a more careful read of a second, different error message, a more careful fix. Still broken, differently again. That's the moment you stop guessing and start actually researching properly instead of pattern-matching to the first error you see.

Turned out Cloudflare's email forwarding has two real, hard limits nobody had hit before: a forwarding rule can only ever have one destination address, and a routing rule can only ever have one action, full stop. There was never a way to declare “forward to this list of people” directly. No combination of config was ever going to work. The actual fix was routing everything through a small Worker that looks up who should get the email and forwards to each of them itself, one at a time, which sidesteps both limits because it's just code running, not a declarative rule fighting its own constraints.

Then the fix itself broke the site's build for an unrelated reason (turns out our fix quietly duplicated a huge chunk of the app into a place it shouldn't have been, and blew past a hard size limit), which is its own small story, and which is exactly the kind of thing that makes “just ship a quick fix” a dangerous sentence to say out loud.

It works now. Properly, not “seems to work.” We know because we went and checked the actual production logs afterwards instead of just hoping.