Optional features should fail open

The feature request was to add a hosted AI classifier to an existing pipeline. My first thought was not about accuracy. It was about what happens the day the model endpoint is slow, or down, or rate-limiting me.

Because here is the thing about bolting an external model onto a pipeline that already works. The pipeline worked yesterday without it. If adding the classifier means the whole request now dies when a third party has a bad afternoon, I have made the system worse, not better.

So I set four rules before writing any of the interesting code.

Off by default, and fail open#

The classifier is opt-in. It ships switched off, and a request only reaches it when a flag says so. Nothing in the default path depends on it existing.

More importantly, when it is on and the model call fails, it fails open. A failure does not stop the pipeline. It logs, returns nothing useful, and the request carries on exactly as it did before the feature existed.

async function classify(input: Input): Promise<Label | null> {
  try {
    return await model.classify(input);
  } catch (err) {
    log.warn("classifier failed, continuing without it", err);
    return null; // pipeline treats null as "no label", not "error"
  }
}

The caller is written to accept null as a normal answer. No label just means no label. A dependency outage degrades the feature, it does not take the request down with it. The classifier is a nice-to-have bolted on the side, not a fuse wired into the mains: when it blows, the lights stay on.

An optional feature that can fail the main path is not optional. It is a new single point of failure hidden behind a feature flag.

A cheap gate in front of the expensive call#

The model call costs money and time. Most inputs do not need it. So before I pay for the expensive call, a cheap heuristic decides whether it is even worth asking.

The heuristic is crude on purpose. It is a fast local check that filters out the obvious cases, the ones where a full model classification would tell me nothing I did not already know. Only the inputs that survive the gate go to the paid endpoint.

if (!cheapHeuristicSaysItMightMatter(input)) {
  return null; // never touched the paid API
}
return await classify(input);

That one gate cut the number of paid calls dramatically, because the expensive path now only runs when the cheap path cannot decide.

Then I cached on top of it. Results are keyed by a hash of the input, so an input I have already classified is free the second time. Repeated inputs, which are common, never hit the model at all.

Write access is guarded harder#

The classifier only reads. But the same job also had a capability that writes to an external system, and write access gets treated with more suspicion than reads.

That path sits behind a default-off flag too, plus a per-tenant rate guard using a fixed window. If the capability is disabled, the endpoint returns 403. If it is enabled but the tenant is over its window, it returns 429. The two cases are different, and the status codes say so.

if (!feature.EnabledFor(tenant)) return Forbid();          // 403, not allowed
if (!rateGuard.TryConsume(tenant)) return TooManyRequests(); // 429, slow down

403 means you cannot do this. 429 means you can, just not this fast. A caller can tell the difference and react correctly, which they cannot do if both look the same.

Optional external features, and AI features especially, should default off and fail open. If a dependency has a bad day, the right outcome is a degraded feature, not a dead request.

Cache aggressively so repeated work is free, and put a cheap gate in front of the expensive call so cost and load fall away when the model is not needed. Guard writes harder than reads, and use status codes that tell the caller which wall they hit. None of this is clever. It is just refusing to let an add-on take down the thing it was added to.