Deduplicate on meaning, not on an id

I had deduplication code that ran on every batch and, as far as I could tell, never removed anything. Two scanners would report the same real issue and both copies would sail straight through.

It was not broken in the way I expected. It was doing exactly what I wrote. I had just told it to compare the wrong thing.

Unique by construction is a silent no-op#

The aggregator collected findings from several sources and was meant to collapse duplicates before storing them. The dedup guard checked each finding’s id against the ids it had already seen.

The problem is that the id was a random UUID generated per object. It is unique by construction. Every finding has a different one, always, even two findings describing the same real problem. So the guard could only ever match an object against itself, which never happens in a set you are building up. It removed nothing.

Two scanners reporting the same vulnerability in the same place produced two different UUIDs, so the guard treated them as unrelated and kept both.

In practice, a security dashboard that should have shown one issue showed the same issue two, three, five times over. Anyone reading it was triaging phantom work and could not trust the counts, which defeats the point of aggregating in the first place.

Deduplicating on a surrogate key that is unique by construction is not a weak check. It is a no-op that only looks like a check.

The fix was to key on meaning instead of identity. I built a semantic fingerprint from the things that actually make two findings the same finding: the category, the normalised location, and a hash of the evidence. Two reports of the same issue now produce the same fingerprint and collapse into one.

// identity is what the finding MEANS, not the id it was handed
const fingerprint = hash([
  finding.category,
  normaliseLocation(finding.location),
  hash(finding.evidence),
].join("|"));

The surrogate id still exists. It is just for referencing a row, not for deciding whether two rows are the same thing.

The same mistake, keyed on a path#

The second version of this bug was hiding in the same project, and it taught me the first fix was not enough on its own.

Records were keyed by filesystem path. On a case-insensitive operating system, C:\X and c:\x are the same directory. As strings they are different. So the same thing got inserted twice, once under each spelling, and the code was perfectly happy because it was comparing strings and the strings differed.

Any key derived from a path, or from any external string, is fragile until you decide its equivalence rules on purpose. Case was the one that bit me. Trailing slashes and mixed separators were waiting their turn.

The fix had three parts. Case-insensitive collation on the lookups so C:\X and c:\x compare equal. Normalise-on-write so a canonical form goes into the store in the first place. And, the part it is tempting to skip, a migration to merge the duplicates I had already created, because fixing the rule going forward does nothing about the mess already sitting in the table.

Both bugs, one shape#

Both bugs are the same shape. Identity is about meaning, and if you dedup on a surrogate or unique key you get a silent no-op that looks like it is working.

Key on a semantic fingerprint, not a generated id. Treat any key built from a path or an external string as untrustworthy until you have decided, explicitly, when two of them count as equal. And when you finally get the rule right, remember to clean up the duplicates the wrong rule already let in.