001 Notes

Perceptual Hashing And Duplicate Review

Perceptual hashing is for duplicate and near-duplicate review, not semantic understanding.

A semantic embedding asks:

Are these concepts related?

A perceptual hash asks:

Does this image still look roughly the same after compression, resize, or small edits?

That distinction matters. A dog sketch and a dog photo may be semantically close, but they are not duplicates. Two JPEG exports of the same dog photo may be visually near-identical even if the bytes differ.

This article explains perceptual hashing as a candidate-generation system: exact hashes, pHash, dHash, wHash, Hamming distance, threshold policy, hash indexes, embedding reranking, ORB verification, and review-first UI behavior.

1. Exact Hash vs Perceptual Hash

A cryptographic file hash is byte identity.

same bytes -> same hash
one byte changes -> completely different hash

That is perfect for exact duplicates.

photo.jpg
photo-copy.jpg

If the files are byte-for-byte identical, a cryptographic hash such as SHA-256 proves it.

But exact hashing fails for visually identical files that have different bytes:

same photo exported at JPEG quality 95
same photo exported at JPEG quality 85
same photo resized from 4000px to 2000px
same screenshot saved as PNG and JPEG

A perceptual hash is visual identity, not byte identity.

same-looking image -> nearby hash
different-looking image -> far hash

The duplicate pipeline needs both:

Tool	Finds	Misses
cryptographic file hash	exact copies	resized or recompressed copies
pHash, dHash, wHash	near visual copies	semantic-only similarity
embeddings	visually or conceptually related images	proof of duplicate identity
ORB/local features	geometric near-duplicates	textureless images and heavy edits

The safe product rule:

Exact hash can prove exact duplication.
Perceptual hash can propose duplicate candidates.
Embedding similarity can rerank candidates.
ORB can verify some geometric matches.
The user decides destructive action.

2. Bit Fingerprint Mental Model

A perceptual hash reduces an image to a fixed-width bit fingerprint.

image A hash: 10110010 01011001
image B hash: 10110010 01011101

Distance is Hamming distance: the number of positions where the bits differ.

A: 10110010 01011001
B: 10110010 01011101
                   ^
Hamming distance = 1

The computation is cheap:

distance = bit_count(hash_a XOR hash_b)

XOR marks differing bits:

hash_a: 10110010
hash_b: 10110110
XOR:    00000100

Then bit_count counts the 1 bits.

Small Hamming distance means the images are likely visually close under that hash method. It does not mean they are definitely duplicates.

3. Why Hashes Survive Small Edits

A perceptual hash intentionally throws away detail.

That sounds bad, but it is the point.

If the hash preserved every pixel perfectly, it would behave like a cryptographic hash and fail after recompression. Instead, perceptual hashes keep coarse information:

brightness structure
large shapes
local gradients
low-frequency layout
scale-level structure

They discard:

small compression noise
minor pixel shifts
some texture detail
some color detail

This is the tradeoff:

More detail kept	Less detail kept
fewer false positives	more robust to small edits
fragile to recompression	more collisions
closer to exact matching	closer to approximate matching

Duplicate review wants approximate matching, so the hash must be lossy.

4. dHash

dHash is difference-based. It captures local brightness direction.

Conceptual algorithm:

1. resize image to 9 x 8
2. convert to grayscale
3. for each row, compare pixel[x] with pixel[x + 1]
4. write 1 if left pixel is brighter, else 0
5. output 8 x 8 = 64 bits

Toy row:

pixels:  [10, 14, 13, 20, 19]
compare: 10 > 14 false -> 0
         14 > 13 true  -> 1
         13 > 20 false -> 0
         20 > 19 true  -> 1
bits:    0101

The full image becomes a grid of gradient-direction comparisons:

0 1 0 1 1 0 0 1
1 1 0 0 0 1 1 0
...

dHash is fast because it needs only resize, grayscale conversion, comparisons, and bit packing.

It works well for:

small resizes
minor compression
similar contrast structure
simple near-duplicates

It struggles with:

rotation
heavy crop
large local edits
watermarks
layout-preserving templates with different meaning

The main intuition: dHash remembers “which direction brightness changes,” not the actual pixels.

5. pHash

pHash is frequency-oriented. The common version is based on DCT-style thinking.

DCT means Discrete Cosine Transform. It represents an image as frequency components:

low frequency:
  broad areas, large shapes, smooth gradients

high frequency:
  edges, texture, noise, fine detail

Conceptual algorithm:

1. resize image
2. convert to grayscale
3. compute frequency coefficients
4. keep low-frequency coefficients
5. compare coefficients to median or average
6. emit bits

Mental model:

pixels
-> broad frequency structure
-> low-frequency block
-> 64-bit fingerprint

pHash often survives JPEG recompression because compression noise mostly affects high-frequency detail. If the broad structure remains, the low-frequency signature stays close.

It is useful for:

recompressed images
resized images
same image with minor noise
same broad composition

It struggles with:

heavy crop
large overlay text
rotation
simple images with similar layout
template collisions

pHash is usually better than exact hashing for near duplicates, but it is not proof.

6. wHash

wHash uses wavelet-style decomposition.

Wavelets describe structure at multiple scales. Instead of only asking “what is the frequency content?” or “which neighboring pixel is brighter?”, a wavelet-style transform separates coarse and finer components.

Conceptual flow:

image
-> coarse structure
-> medium structure
-> fine structure
-> compact low-scale signature

Why keep wHash alongside pHash and dHash?

Hash	Signal
dHash	local brightness direction
pHash	low-frequency composition
wHash	multi-scale coarse structure

Each hash fails differently. A candidate that is close under all three signals is stronger than a candidate that barely passes one hash.

The product should store the matched hash type:

matched by pHash distance 4
matched by dHash distance 6
matched by wHash distance 5

Without that evidence, “duplicate” becomes a black box.

7. Hamming Distance Thresholds

Thresholds are policy, not truth.

For a 64-bit perceptual hash:

distance 0:
  identical hash

distance 1-4:
  very close

distance 5-8:
  common near-duplicate range

distance 9-16:
  broader candidate search, more false positives

distance > 16:
  often too loose unless used with other filters

These ranges are not universal. They depend on:

hash algorithm
image type
dataset size
review tolerance
whether a reranker follows

A strict duplicate cleanup workflow should use lower thresholds. A “find things that might be related” workflow can use higher thresholds because the user expects exploration.

The UI should show distances:

Candidate A:
  pHash distance: 2
  dHash distance: 4
  wHash distance: 3

Candidate B:
  pHash distance: 12
  dHash distance: 15
  wHash distance: 11

Those are not equally trustworthy. The user should be able to see that.

8. Index Design

A duplicate tool should not recompute hashes for every query.

Store hash records with invalidation metadata:

Field	Purpose
image path or image id	maps hash to file
mtime_ns	detects changed file
file_size	detects changed file
pHash hex	readable hash
dHash hex	readable hash
wHash hex	readable hash
pHash u64	fast XOR and bit count
dHash u64	fast XOR and bit count
wHash u64	fast XOR and bit count

Index update flow:

flowchart TD
  A[Scan image paths] --> B{Path in index?}
  B -- no --> C[Compute pHash/dHash/wHash]
  B -- yes --> D{mtime or size changed?}
  D -- yes --> C
  D -- no --> E[Reuse stored hashes]
  C --> F[(Hash index)]
  E --> F
  F --> G[Prune removed paths]

The invalidation rule matters. A stale hash can produce wrong duplicate candidates after the file is edited.

For a local desktop app, SQLite is a practical default:

simple deployment
transactional updates
easy pruning
easy inspection
works without a server

Start with the simplest index that is correct.

9. Querying The Index

Basic query:

query_hashes = compute_hashes(query_image)

for candidate in hash_index:
  phash_distance = bit_count(query.phash XOR candidate.phash)
  dhash_distance = bit_count(query.dhash XOR candidate.dhash)
  whash_distance = bit_count(query.whash XOR candidate.whash)
  best_distance = min(phash_distance, dhash_distance, whash_distance)

  if best_distance <= threshold:
    emit candidate

Worked example:

Candidate	pHash dist	dHash dist	wHash dist	Pass threshold 8?
A	2	4	3	yes
B	7	12	9	yes, pHash
C	18	21	17	no
D	5	6	5	yes

Sorting should prefer:

lowest best distance
more matching hash families
exact file hash matches first
same dimensions when useful
larger source-quality files when choosing representatives

The query can also accept a candidate set:

only current folder
only selected tags
only visible filtered results
only images from the same scan session

Filtering before distance scanning is often a bigger speed win than adding complicated indexing.

10. Hash Prefilter Plus Rerank

Perceptual hashes are cheap candidate generators.

Embeddings and feature matching are heavier. They should usually run after the hash prefilter:

flowchart LR
  A[All images] --> B[Perceptual hash threshold]
  B --> C[Candidate set]
  C --> D[Embedding rerank]
  D --> E[Optional ORB verification]
  E --> F[Review group]

Embedding rerank answers:

Among the hash candidates, which images are closest in visual or semantic vector space?

This is useful because a hash threshold can be noisy. Embeddings can move the best candidates to the top of the review group.

But embeddings should not be treated as duplicate proof. They are a ranking signal.

Good policy:

hash -> generate candidates
embedding -> rank candidates
ORB -> verify geometry where useful
human review -> approve action

11. ORB Verification

ORB stands for Oriented FAST and Rotated BRIEF. Conceptually, it detects local keypoints and describes the area around them with compact binary descriptors.

ORB flow:

1. detect keypoints in image A
2. detect keypoints in image B
3. compute descriptors around keypoints
4. match descriptors between images
5. filter weak matches
6. optionally estimate geometry

Keypoints are local visual landmarks:

corners
distinct texture patches
small repeatable structures

Descriptors let the system compare those landmarks.

ORB helps when:

same image is cropped
same image is resized
same object appears with mild transformation
same scene has local feature overlap

ORB struggles when:

image is textureless
screenshot has flat regions
patterns repeat many times
crop removes most landmarks
image is heavily edited

ORB should not be the first pass across a huge folder. It is better as verification after cheaper filters.

12. Duplicate Groups Are Review Objects

A duplicate group is not a command to delete.

A useful group contains evidence:

representative image
candidate images
exact hash status
perceptual hash distances
matching hash family
embedding similarity if computed
ORB match score if computed
file size
dimensions
modified time
paths
preview thumbnails

The UI should support safe actions:

keep largest
keep newest
keep selected
open containing folder
move candidates to review folder
mark as not duplicate
hide group
rescan changed files

Automatic deletion should be avoided unless the evidence is exact byte identity or the user explicitly chooses it.

Near-duplicate evidence is not enough for destructive cleanup.

13. Failure Modes

Perceptual hashing fails in predictable ways.

Case	Problem
crop	global structure changes
rotation	many hashes are orientation-sensitive
watermark or text overlay	structure changes
screenshots	flat regions collide
memes/templates	same layout, different meaning
burst photos	visually close but not duplicates
resized thumbnails	detail loss can create false matches
generated images	similar composition can collide

The product should communicate uncertainty:

near duplicate candidate
possible duplicate
matched by pHash distance 5
needs review

It should avoid certainty:

duplicate
safe to delete
same image

unless exact-file evidence exists.

14. Performance And UI Runtime

The first optimization is caching.

The second optimization is not doing work on the UI thread.

Practical rules:

compute hashes in worker jobs
store hashes persistently
reuse unchanged rows
use u64 XOR and bit_count
batch database writes
filter candidate paths before full scans
show progress
support cancellation
surface index size and cleanup controls

For larger indexes, possible next steps:

bucket by hash prefix
use multiple hash tables
restrict search to active folder/filter
parallelize candidate scanning
use native popcount/vectorized operations
add locality-sensitive hashing only after profiling

Do not start with a distributed vector database or complicated indexing layer. A SQLite table plus u64 XOR is understandable, portable, and often fast enough.

15. What Was Used, Removed, Or Deferred

Used:

pHash
dHash
wHash
SQLite-backed hash index
mtime and size invalidation
u64 XOR plus bit_count distance
duplicate candidate grouping
embedding rerank
optional ORB verification
review-first UI policy

Removed or avoided:

treating perceptual hashes as semantic labels
hiding hash distance
automatic deletion from near-duplicate evidence
recomputing hashes unnecessarily
blocking the UI thread
presenting candidate matches as certain duplicates

Deferred:

GPU hash kernels
large-scale locality-sensitive hashing
rotation-invariant duplicate proof
fully automatic duplicate cleanup
destructive cleanup based only on embeddings

The key idea: perceptual hashing is a candidate generator. It is cheap, useful, and explainable, but it should feed a review workflow rather than act as an authority.