Graceful drain on shutdown: the deploy that killed a three-minute job

Someone shipped a small change at the wrong moment. Nothing dramatic, a normal deploy, the kind that happens ten times a week. Except this one landed while a render was three minutes into a four minute job.

The deploy restarted the process. The job died where it stood. What was left behind was a half written output file and a job the queue still believed was active, because nothing had told it otherwise.

A restart is not a clean stop#

Here is the thing I had underestimated. A deploy, a scale event, a crash recovery, they all boil down to the same move: the platform sends your process a SIGTERM and, if you do not react, a SIGKILL shortly after.

If your worker just exits when that signal arrives, it exits out from under whatever it was doing. A quick job barely notices. A multi-minute render notices a lot.

The job was mid write when the process vanished. The output was partial, so anything downstream that trusted it was now working from a corrupt file. And because the worker never got to report back, the queue had no idea the job had stopped. It sat there marked active, occupying a slot, waiting for a worker that was already gone.

From the customer’s side the story is simple and maddening. They waited out a four minute render and got a broken file, while the system still insisted the work was in progress, so nothing retried it and nothing told them it had died.

A restart is not the enemy, an ungraceful one is.

Let close() finish the work#

The fix is to treat SIGTERM as “start wrapping up”, not “die now”.

The worker installs handlers for SIGTERM and SIGINT. When one fires, instead of exiting, it calls close() on each worker. That call is the important bit: close() stops the worker from picking up new jobs and then waits for the in-flight job to finish. Only once the work is genuinely done does the process close its connections and exit.

So the sequence during a deploy becomes: signal arrives, stop taking new work, let the render run to completion, write the output fully, mark the job complete, then shut down. The deploy waits those extra sixty or ninety seconds, and everyone is better for it.

The half written files stopped appearing. The stuck active jobs stopped appearing. And a rolling deploy became genuinely safe, because each old worker drains its current job before it steps aside for the new one.

There is a detail worth naming. Platforms give you a grace period between SIGTERM and SIGKILL, and it has a limit. If your jobs can run for minutes, make sure that grace window is long enough to cover a job finishing, otherwise you are back to being killed mid-run, just politely this time.

The honest takeaway: if a worker holds long jobs, graceful drain on SIGTERM is what turns routine restarts and rolling deploys from a gamble into a non-event. You are not trying to avoid restarts, you cannot, they are how deploys work. You are trying to make them boring.

I still double check the grace period every time I touch the deploy config, because that is the one number that quietly undoes all of this. But the principle holds: do not exit out from under an active job, let close() finish the work first.