The config key that ran production on a disposable database

This is a short one, and it is the kind of bug that makes you go very quiet when you find it. Picture a shop taking real payments into a till that empties itself every night, with nobody noticing until the morning the money is gone.

An application read its database connection from a configuration key. Call it ConnectionStrings:Primary. The production deployment manifest supplied the connection string under a different key, the one a slightly older convention used. The two names did not match.

Nothing errored. Configuration systems are famously relaxed about this: a key that is not found is not an exception, it is just absent. So the real database connection string was silently ignored, the app fell through to its fallback, and the fallback was a local SQLite file inside the container.

Which means “production” was running on a disposable database that lived inside the container and vanished every time it restarted. No connection error, no warning at boot, no red anything. Just a production system quietly writing its data somewhere it would lose it, waiting for the first restart to make the problem visible in the worst possible way.

Why it is so quiet#

Two failure modes stacked up here, and both are silent by design.

Configuration binding fails soft. A misspelled or mismatched key does not blow up; it produces a default, an empty string, a null, a fallback. That is convenient right up until the missing value was load-bearing.

And a fallback database is worse than no database, because no database at least fails loudly. A fallback comes up, accepts writes, and behaves like it is working. The system is not broken in any way you can see. It is broken in a way you find out about later.

A config key is a contract between your app and your deployment. Nobody type-checks it, nobody enforces it, and it fails silent when it breaks. So you have to enforce it yourself.

The fix is to be loud on purpose#

The fix is not really about this one key. It is a policy: assert your critical configuration at startup and refuse to boot without it. If the expected database provider or connection is not actually present, throw, and make the process fail to start. Loudly, immediately, at deploy time, when someone is watching, rather than quietly, later, when the container recycles and the data is gone.

A convenient fallback is a fine thing for local development. In production it is a trap, because it turns “you forgot to configure me” from an obvious crash into an invisible time bomb. The whole art here is refusing the convenience exactly where it would hurt: fail fast, treat the config keys as a contract you verify end to end, and never let the app come up pretending everything is fine on a database that is about to disappear.