One process, eleven workers: how a heavy render starved the cheap jobs

I opened the worker file expecting a bit of a mess. What I found was eleven queue workers, all created in the same process, sharing one event loop, and not a single one of them configured on purpose.

It ran fine in a demo. It fell apart the first time a real render showed up.

Eleven tenants, one event loop#

The pipeline had lots of little jobs (accept an upload, schedule a task, tidy up) and a few very heavy ones (a CPU and GPU render that pinned a core for minutes). Every one of those got its own worker, and every worker lived in the same process.

That is the trap. A heavy job that saturates the event loop does not just slow itself down, it starves everything else in the process. While the render churned, the cheap upload and schedule jobs, work that should take milliseconds, sat waiting for a turn on a loop that was busy.

And it was fragile in the other direction too. One unhandled error in any worker took down the whole process, so a bug in the trivial cleanup job could kill the render that had been running for three minutes.

Heavy and light do not belong in the same process. Heavy starves light, and any crash is shared. To a user that looks like an upload spinning forever while someone else’s video renders, and every so often the whole thing falling over and taking the in-flight work down with it.

Nobody set the knobs#

The second problem was quieter and, honestly, more embarrassing. None of the eleven queues set concurrency, attempts, backoff, or retention. They were all running on library defaults.

The defaults are almost never what you want. Concurrency defaulted to one, so even where parallelism was safe we processed one job at a time. Attempts defaulted to no retries, so a single transient blip (a network hiccup, a locked file) failed the job permanently with nowhere to recover.

The defaults are a placeholder for a decision you have not made yet, not a sensible setting someone chose for you.

Then there was the slow leak. Completed jobs kept their full payloads in Redis, and nothing ever removed them. Every job that ran left its data behind forever. Memory climbed all day, and the only thing that reset it was a restart, which of course looked like the fix and hid the real cause for weeks.

Split the workloads, set the knobs#

The fix came in two parts.

First, split heavy from light. The render moved into its own worker process, on its own schedule, where it can pin a core without touching anything else. The cheap jobs stayed together in a separate process where they belong. Now a stuck render cannot starve an upload, and a crash in one does not take the other down.

Second, set every knob per queue, on purpose. Concurrency to match what the work actually tolerates. Attempts with exponential backoff so a transient failure gets a second and third try instead of dying on the first. And retention, removeOnComplete and removeOnFail, so finished jobs get cleaned up instead of piling into Redis until the box falls over.

The honest takeaway: the eleven workers were never really the problem, the missing decisions were. Splitting heavy and light stopped the starving. Setting attempts and retention on each queue turned a fragile, leaking pipeline into a boring one. Boring is the goal.

I still have queues where I am guessing at the right concurrency and correcting later. That is fine. Guessing on purpose and writing the number down beats inheriting a default you never chose.