Glossary

Duplicate detection

Also called: deduplication

Duplicate detection identifies documents whose content is already present in the repository, normally by comparing a cryptographic hash of the file. Filename comparison misses the common case entirely, because the same document has usually been renamed somewhere between the two copies.

Why hashing rather than filenames

The same document arrives twice as "invoice.pdf" and "INV-4471 scan.pdf". A filename comparison finds nothing; a hash comparison finds an exact match regardless of what it has been called. Hashing also catches the case where a file is re-uploaded years later by someone who did not know it was already there.

What hashing cannot catch is the same document arriving as a scan and as a native PDF — different bytes, same content. That case needs metadata matching, typically supplier plus document number.

The real reason it matters

Storage cost is the obvious argument and the less important one. The document control problem is that a duplicate is a second copy with an independent life: it does not receive the new version, it is not covered by the retention rule applied to the original, and it circulates as current after the original has been superseded.

A duplicate report is therefore a document control report, not a housekeeping one.

Flag rather than block

There are legitimate reasons to hold the same bytes twice — a contract filed under both parties, a certificate relevant to two assets. Blocking the upload forces a workaround; flagging it and recording the decision preserves the information.

FAQ