From Pixels and Words to Semantic Search

Compare a red Ferrari, a red apple, and a blue Porsche.

A colour representation may put the Ferrari near the apple because both are red. A shape-and-subject representation may put the Ferrari near the Porsche because both are sports cars. Neither distance calculation can repair a representation that retained the wrong information.

In this tiny experiment, a red six-pixel shape represents a car. Treat every red cell as 1 and every background cell as 0. Switch the candidate among three states and watch what the fixed-coordinate comparison reports.

After the one-pixel shift, a person still sees the same car. Raw comparison sees four disagreements: two addresses where red disappeared and two new addresses where red appeared. It has no rule saying “look one cell to the right.”

The faded car exposes an even stranger result. Its six pixels changed from 1.0 to 0.6, giving a squared distance of 0.96. That is smaller than the shifted car’s distance of 4.00. Raw distance therefore considers the faded, altered car closer than the unchanged car that merely moved.

We can reduce the animation to two grayscale pixels. Here 0 means black and 1 means white:

Squared L2 distance is:

Now compare an altered gray pattern whose light and dark values remain at the original coordinates:

Raw distance prefers the unrelated aligned pattern over the shifted original. The numbers are tiny, but the failure mechanism scales to real images.

A detailed representation is not automatically a useful similarity representation.

A vector is merely an ordered list of numbers. Raw pixels, random numbers, and handwritten measurements can all be vectors. We use the word embedding when an object has been mapped into a space whose geometry carries learned relationships.

The last column is the important one. A database can store any of these vectors, but storage does not make their distances meaningful.

There is no objective-free meaning of similarity. Use the experiment below to train the same illustrative images for two different relationships.

Under subject training, the red car approaches the blue car even though their colours differ. Under colour training, the same red car approaches the red apple even though their subjects differ.

Nothing about Euclidean distance or cosine similarity chose this behaviour. Training chose the representation on which those distance calculations operate.

The dimension is the number of coordinates in each embedding. A 32-dimensional embedding contains 32 numbers; a 512-dimensional embedding contains 512.

More dimensions give the model room to preserve more independent patterns, but they also increase memory, storage, transfer, and comparison work. More is not automatically better: unused or poorly trained dimensions can be redundant or noisy.

Use the dimension lab to see the systems cost for float32 embeddings. Each float32 coordinate requires four bytes.

To test dimension rather than guess, hold the dataset, splits, preprocessing, model family, training budget, and evaluation procedure constant. Change the dimension and measure retrieval quality, latency, and memory together.

Suppose the hidden query is a red Ferrari. Compare the five nearest neighbours under the two spaces from the training experiment:

The subject-trained space answers “what kind of object is this?” The colour-trained space answers “what has a similar dominant colour?” Both can be internally consistent. Only the application tells us which is useful.

Hide each labelled image in turn, ask its five nearest neighbours to vote, and count correct predictions:

Nearest-neighbour accuracy is only one probe. A real evaluation should also inspect failures, class balance, duplicates, out-of-distribution images, retrieval at several values of K, latency, and memory.

The central lesson is:

An embedding does not contain universal meaning. It contains geometry learned under a particular training contract, and that geometry earns trust through experiments.

The lookup table initially gives bank the same learned token vector wherever it appears. Self-attention creates a context-sensitive update.

For one attention head, use this beginner-sized mental model:

query from "bank" asks: which positions are useful to me?
key from every token answers: what kind of information do I contain?
value from every token carries: the information that can be mixed

The query–key comparisons become attention scores. A softmax converts those scores into non-negative weights that add to 1. The output for bank is a weighted mixture of the value vectors.

In the river sentence, the teaching weights are:

The prime in bank′ means “the updated representation of bank.” Each V is a complete value vector, not one scalar. The weighted additions happen coordinate by coordinate across that vector.

Switch to open a bank account. The lookup vector for bank begins the same, but the 70% weight on account produces a different contextual update. This is how one token can participate in two meanings without requiring two token IDs.

This does not mean a single attention head is a complete explanation of meaning. Real transformers use many heads and layers, residual connections, normalization, and nonlinear transformations. The dry run isolates the central intuition: a token representation can depend on the entire sentence.

A tensor ledger is bookkeeping for the arrays moving through the model. At every stage it records:

what the numbers currently represent
how many axes the array has
what each axis counts
which axis appeared or disappeared

The shape does not reveal the values inside the tensor. It describes how those values are arranged.

For text, three axis names appear repeatedly:

Therefore:

We will dry-run a tiny model with only eight token positions and four hidden features. The smaller numbers let us draw every part. Afterwards we will map the same operations to a realistic [1, 77, 768] tensor.

It is 1 only because we embedded one sentence. If we process:

sentence A = "a red sports car"
sentence B = "waves on a beach"
sentence C = "a photographed invoice"

the token-ID shape becomes [3, 77], the transformer output becomes [3, 77, 768], and the final embedding matrix becomes [3, 512].

row 0 → embedding for sentence A
row 1 → embedding for sentence B
row 2 → embedding for sentence C

Batching adds more rows. It does not mix the three sentences into one embedding.

Use these as tests, not assumptions:

1. Paraphrase: compare red sports car with crimson performance automobile. A useful semantic model should often keep them near despite low word overlap.
2. Order: compare dog bites person with person bites dog. If the model ignores the changed relationship, its representation is too coarse for that task.
3. Negation: compare a red car with not a red car. Embeddings can be weak at strict logical negation even when they capture the subject strongly.
4. Truncation: place the important phrase after the maximum context length. If the tokenizer removes it, the encoder never sees it.

The final text vector is a learned summary, not a lossless compressed sentence. It may preserve subject, style, action, and broad intent while losing exact numbers, spelling, word order subtleties, or logical operators.

Use at least three clear subjects and one altered copy:

And use text probes with deliberate controls:

T1 = "a red sports car"          expected: A or B
T2 = "waves on a sandy beach"   expected: C
T3 = "a photographed receipt"   expected: D
T4 = "a blue bicycle"           expected: none strongly

The altered copy checks invariance. The unrelated images check separation. The unsupported bicycle query checks whether the system still returns a top result even when no result is genuinely good.

image_tensor = image_preprocess(image).unsqueeze(0)
token_ids = tokenizer(["a red sports car"])

print(image_tensor.shape)  # [1, 3, 224, 224], for this example model
print(token_ids.shape)     # [1, 77], for this example tokenizer

At this point:

image_tensor = normalized pixels
token_ids    = integer vocabulary addresses

Neither is a semantic embedding yet.

Print structural facts:

image tensor shape:       [1, 3, 224, 224]
token ID shape:           [1, 77]
image feature shape:      [1, 512]
text feature shape:       [1, 512]
image feature dtype:      float32
image embedding norm:     1.0000
text embedding norm:      1.0000
all values finite:        true

Each check catches a different class of error:

A vector can pass every structural check and still be semantically poor. The next stage tests behaviour.

Create controlled variants of A:

For each variant, record:

similarity(original, variant)
rank for "a red sports car"
rank for an unrelated control prompt

This turns “the embeddings look good” into evidence. It also reveals what the chosen representation treats as invariant.

Persisting only the path and vector is unsafe:

IMG_1042.jpg → [0.12, −0.34, ...]

The cache needs the source identity and embedding recipe:

image path
file size
modification time or content hash
model name
weight revision
preprocessing revision
embedding dimension
dtype
normalization rule
vector bytes

If the file changes, re-embed it. If the model or preprocessing changes, re-embed the whole affected space. Old and new readable arrays may be geometrically incompatible.

The harness calls ClusterLens’s own EmbeddingService. It records the model ID, local cache revision, preprocessing signature, input size, output dimension, package versions, dataset indices, image checksums, seed, and timing trials. The model weights are cached locally and the final run forbids model downloads.

For every controlled image, create five variants:

original
├── JPEG quality 35
├── shrink to 64 × 64, then restore
├── brightness × 0.65
├── centre crop to 70%, then restore
└── horizontal flip

The base and altered images are encoded independently. Their normalized vectors are compared with cosine similarity. A score near 1 means the transformation changed the direction only slightly. That is evidence of invariance—not proof that the model is correct about every image.

Actual run

ModelCLIPSigLIPDINO TransformationJPEG quality 3564px downscale65% brightness70% centre crophorizontal flip Fixture

cosine——

Mean over all 12—

Lowest of 12—

Same-class text top-1—

—

The horizontal flip is an especially useful control. A semantic organizer often wants a flipped bicycle to remain a bicycle; an application that cares about left-versus-right direction may not. “Invariant” is never automatically good. It is good only when the ignored change is irrelevant to the task.

For CLIP and SigLIP, five prompt templates are embedded per CIFAR-10 class. The five vectors are normalized, averaged into one class prototype, and normalized again. Every image is assigned to its highest-scoring text prototype.

This is the cleanest practical distinction between “an image embedding” and “a shared image–text embedding.” DINO still produces vectors, but there is no DINO text vector to compare with them in this contract.

With the correct tokenizer contracts, the 500-image zero-shot run reached 90.4% for CLIP and 93.0% for SigLIP. Those results are not trained CIFAR classifiers: the only class definitions were the ten groups of five text prompts.

The experiment caught a real SigLIP preprocessing bug

The first run produced a suspicious combination:

SigLIP image 5-NN accuracy:     85.0%
SigLIP text zero-shot accuracy: 10.6%

Ten CIFAR classes make 10% approximately chance. The image encoder was clearly useful, so “SigLIP is bad” was a weak explanation. We inspected the exact text tensor contract.

ClusterLens had asked the tokenizer for padding=True. That pads every prompt only to the longest sequence in the current batch. This SigLIP checkpoint was trained with a fixed 64-position text sequence. Its pooling head depends on that layout, so changing the sequence length changed the representation before any image–text comparison occurred.

We changed the SigLIP text path to use fixed maximum-length padding and truncation, then reran all 500 images with the same weights, prompts, indices, and seed. Accuracy rose to 93.0%. A regression test now asserts that SigLIP receives 64 token positions.

This is why “use the model’s preprocessing” is not ceremonial advice. A tensor can have valid integers, inference can complete, vectors can be finite and unit-normalized—and the semantic contract can still be wrong.

A top-K search always returns something. A threshold can reject weak pairs, but where should the boundary go? The controlled set provides eighteen same-class base-image pairs and forty-eight different-class pairs for each model.

The slider is not offering a universal threshold. It demonstrates why one does not exist: the score distributions depend on the model, dataset, and definition of relevance. A threshold must be calibrated on representative labelled data.

Each value is the median of three twelve-image CPU runs after one excluded warm-up. It includes image preparation and inference through the same service path used by ClusterLens. It is suitable for comparing these runs on this machine, not for claiming a universal model speed. The observed medians were 30.8 ms/image for CLIP, 81.9 ms/image for SigLIP, and 31.9 ms/image for DINO.

Loading measured artifact…

The evidence file is bundled into this page, not hidden behind a separate dashboard. Its checksums and selected CIFAR indices let another run verify that the same inputs were used. The command that regenerates it is:

.venv/bin/python scripts/semantic_search_evidence.py \
  --models clip siglip dino \
  --samples-per-class 50 \
  --seed 20260901 \
  --offline

This section still has limits. Twelve controlled images are enough to expose a mechanism, not establish broad robustness. CIFAR-10 is public and reproducible, but tiny and unlike a real photo library. The honest conclusion is therefore not “model X wins.” It is that the representation contract, invariance, neighbour quality, text alignment, compression loss, threshold errors, and latency can all be measured instead of guessed.