Making peace with at-least-once: field notes from a message queue

I have been building an event-driven service lately: a pile of background workers pulling messages off a durable queue and doing things with them. The queue gives an at-least-once guarantee, which is a polite way of saying it will occasionally hand you the same message twice, and there is nothing you can do to stop it.

You can fight that, or you can design for it. Fighting it goes badly. Here are four things I changed once I gave up and built for the guarantee I actually had.

The consumer that could not be poisoned#

Early on, one malformed message could take down a whole worker. It would fail to deserialise, throw, and the consume loop would die with it. One bad message, whole pipeline wedged. That is a poison pill, and it is a genuinely nasty outage because it looks like the whole system broke when really one event was shaped wrong.

The fix was to separate two failures that feel similar and need opposite responses:

• The message is garbage (it does not match the schema): drop it and keep going. Nak, move on. One bad event should never block the good ones behind it.
• My handler failed (the message was fine, my code fell over): let it retry.

So every message now gets validated against a typed schema before the handler runs. Schema failure means nak-and-continue. Only a clean run through the handler acks the message. The loop stays alive no matter what turns up on the wire.

“The message is broken” and “my code is broken” are different incidents. If you handle them the same way, one of them is being handled wrong.

At-least-once is a promise you keep downstream#

Once you accept the same message might arrive twice, the correctness has to live in the writes, not the delivery. Every write that a message triggers has to be safe to run twice.

In practice that meant keying everything on a stable natural identity rather than “insert a new row.” An upsert keyed on (source, external_id). Downstream records keyed on the id of the thing that caused them. Process the same event twice and the second pass is a no-op, not a duplicate. Skip that and a redelivery becomes a repeated action out in the world: the same order placed twice, the same balance credited twice, off a duplicate the queue was always entitled to send.

It is a small mental shift with a big payoff: redelivery stops being a bug you chase and becomes a thing that just happens and does not matter.

The async call that was quietly blocking#

Then a sweep for blocking calls in async code turned up twenty-odd of them. Most were harmless: schema migrations legitimately run on a sync engine at startup, who cares. But one was on a hot path, and it was uploading to object storage like this:

await asyncio.to_thread(client.put_object, ...)

That looks async. It is not really. to_thread takes a blocking call and parks it on the event loop’s shared thread pool. Fine for something that runs once at boot. On a hot path, under load, every one of those uploads is competing for a bounded pool, and you have quietly reintroduced the blocking you thought you had removed, plus backpressure you cannot see.

Swapping it for a genuinely async client fixed it. The rule I kept: to_thread is for occasional setup work, not for anything that runs per request.

A test that says “never”#

The riskiest path in the whole system does something irreversible at the end of a chain of safety checks. Example tests felt thin there. “I checked the cases I thought of” is exactly the wrong reassurance for code where being wrong is expensive and permanent.

So instead of listing cases, I wrote down the one rule that must always hold, and let a property-based test hammer it with a thousand generated inputs: anything the safety checks classify as “reject” must never, ever reach the irreversible step. Not “does not in these five examples.” Never, across the whole generated input space.

That is where property tests earn their place. When you can name an invariant that has to hold for all inputs, a machine generating adversarial inputs will find the hole you did not think to look for. And the same instinct shows up everywhere once you have it: enforce the hard limit server-side, re-reject it with an audit trail, never trust that the layer above got it right.

The through-line#

None of these are exotic. They are all the same move: stop assuming the happy path, and design for the guarantee you actually have instead of the one you wish you had. The queue will double-deliver. A message will be malformed. The load will find your one blocking call. Something will try to do the irreversible thing it was not allowed to.

Build like all of that is going to happen, because it is. That is not pessimism. It is just reading the fine print.