A long job behind a sync request: why the socket died at 60 seconds

The bug report said the endpoint was “flaky”. Sometimes it worked, sometimes the caller got a truncated response, sometimes a raw connection error and no body at all.

The truth was worse than flaky. The endpoint was doing minutes of real work behind a single synchronous request, and the network was quietly deciding how long it was allowed to take.

Three heavy stages, one open socket#

Here is what the handler actually did. A client sent a POST, and inside that one request the service ran three heavy stages back to back: decode and prepare the input, run a GPU render, then encode the output. On a normal file that was two to four minutes of work.

The whole time, the caller sat there holding an open socket, waiting for a response body that would not arrive until all three stages finished.

That is the part that breaks. A browser, a load balancer, and a reverse proxy all have idle timeouts, and the common defaults sit around thirty to sixty seconds. Nothing was flowing back down that socket during the render, so as far as the proxy was concerned the connection was idle. At the timeout it cut the cord.

The client did not see “still working”. It saw a truncated response or an ECONNRESET, which reads like a crash even though the server was busy doing exactly what it was told. From the person’s side the upload looked like it had failed, so they tried again, and each retry quietly kicked off another few minutes of GPU work the server would finish and nobody would collect.

The wrong theory, then the root cause#

My first theory was memory. Long GPU jobs, big buffers, surely something was falling over under load. I added logging around the render stage and watched a job run to completion in the server logs, output written, exit clean, while the client had already recorded a failure ninety seconds earlier.

That was the tell. The server finished the work. The connection between us had died long before, at a suspiciously round number of seconds.

The root cause was never the render. It was the shape of the request. We had put a minutes-long operation behind a protocol that everything in the path assumes is quick.

A response that takes minutes is not a slow request, it is a long job forced down a protocol built for something quick.

Return a handle, do the work behind a queue#

The fix is a shape change, not a speed change. Anything that can run for minutes should not block the caller.

You accept the request, enqueue the work, and return a handle straight away, an id and a status of queued. The heavy stages run behind a queue, out of the request path. The caller polls a status endpoint, or you stream progress back, and pulls the result when the job reports done.

Now no single socket has to stay open for the length of the job. The POST returns in milliseconds. The proxy timeout never fires because there is nothing idle to time out. And a client that disconnects does not kill the work, because the work was never tied to that connection in the first place.

The honest takeaway: a timeout at a round number like sixty seconds is rarely about your code being slow. It is the network telling you that you picked the wrong shape. If an operation can take minutes, give the caller a receipt, not a held breath, and do the real work behind the queue.

I am still tuning how much progress to stream versus just letting people poll. But the category of bug, the socket that dies mid render, is gone, and it went the moment the job stopped pretending to be a request.