Deterministic by name, random at runtime

A unit test was failing about one run in five. It asserted that a component returned the same vector for the same fixed input, and the class was right there in the name: deterministic. Fixed inputs, deterministic component, intermittent failure. Something did not add up, and “flaky test, run it again” was the wrong conclusion.

The hash that reshuffles on every boot#

The component turned a piece of text into a numeric vector, and to do that it hashed each token with the language’s built-in string hash, GetHashCode. That seems harmless. It is not, because in this runtime the built-in string hash is randomised per process. Different seed every time the process starts. It is a deliberate hardening measure, meant to stop attackers from engineering hash collisions to flood a dictionary.

Which means the vectors were perfectly stable within a single run, and completely different after a restart. The test only caught it because the runner occasionally reused a process and occasionally did not, so the values sometimes matched and sometimes did not. One in five.

It is like relabelling every drawer in a filing cabinet each morning. Everything you filed yesterday is still in there, but the map you used to find it no longer matches, so your searches quietly start coming back with the wrong drawer.

The real problem was much bigger than a test. Any vector persisted to storage would silently stop matching a vector recomputed after the next deploy. Similarity search would quietly degrade with no error, no crash, just slowly worse results that nobody could explain. The intermittent test was the only visible symptom of a correctness bug that would otherwise have surfaced as a vague “search feels off” months later.

The fix was to stop using the runtime hash for anything that has to be stable and use an explicit, well-defined one instead (FNV-1a, in this case), so the same input produces the same number on any machine, in any process, forever.

Two lessons, both sharper than they look#

First: “deterministic” has to mean deterministic across processes and machines, not just within one run. Anything you persist, compare across restarts, or shard on, must not be derived from a runtime-seeded hash. The built-in string hash is for in-memory dictionaries that live and die inside one process, and nothing else.

If a value crosses a process boundary, in storage, over a wire, or just across a restart, it cannot come from a hash that reseeds itself. “Same within one run” is not the same as “same.”

Second, and this is the one I keep relearning: a test that only fails under repetition is often not flaky. It is pointing at a real invariant that only breaks sometimes. The instinct to re-run it until it goes green is the instinct to delete your only evidence. The one-in-five failure was the system telling me the truth about itself, and the right response was to listen, not to retry.