Expensive handles want to be singletons

Building an image pipeline that ran machine-learning models, I hit an old lesson in an unfamiliar setting. Some objects are cheap to make and some are expensive, and the expensive ones do not belong in the per-request lifecycle no matter how tidy that feels.

What “expensive” actually means here#

The pipeline used neural models for detecting and recognising faces. Each model is loaded through an inference runtime that wraps a native session: it reads weights off disk, allocates buffers outside the managed heap, and hands you a handle. That handle is the thing you call to run inference. Creating it is slow, measured in the time it takes to load a model, and the memory it holds is native, so the garbage collector cannot clean it up for you if you forget to dispose it.

Now imagine registering that with the default “one per request” lifetime, the way you would a lightweight service. Every single request reloads the model from disk before doing any work, so you pay the load cost on every call, and every request that fails to dispose its session leaks native memory the runtime cannot reclaim. Under any real load that is a slow crash.

It is the software equivalent of re-reading the entire manual from page one before answering each phone call, and never throwing the old copies away, so every caller waits longer than they should while the desk slowly buries itself and tips over.

The shape that works#

So the models are registered as singletons: created once, kept alive for the life of the process, shared across every request. The comment in the code is literally “keep the sessions alive,” because that is the whole point. In front of them sits a per-request pipeline object that is stateless and cheap, and orchestrates calls into the shared models.

That split is the pattern worth keeping:

• The expensive, stateful handle (the model, a database connection factory, an image-processing context, anything that loads or opens something native) is a long-lived singleton.
• The cheap, stateless facade in front of it takes the per-request lifetime, and just coordinates.

If creating it loads a file, opens a socket, or allocates outside the managed heap, it is not a per-request object. Make it live once, and put a disposable, stateless thing in front of it.

The general instinct#

This is not really about machine learning. It is the same reasoning behind connection pooling, behind reusing an HTTP client instead of newing one per call, behind keeping a compiled regex or a template engine around. The cost you cannot see in a single call, a few hundred milliseconds of setup, a chunk of native memory, becomes the whole story once that call happens thousands of times.

The tell is in the constructor. If making the object does real work, reads, opens, compiles, loads, then its lifetime is a decision you have to make on purpose, and “new one every time” is almost never the right answer. Cheap things can be careless about lifetime. Expensive things cannot, and the runtime will not warn you which is which. That is your job to know.