Semantic Image Search From Embeddings To Ranked Results

A normal search asks direct metadata questions:

does filename contain "receipt"?
does folder name contain "invoice"?
was this image modified last week?

Semantic search asks a different question:

which images are visually or conceptually close to the query?

The query can be another image:

find images like this one

Or it can be text:

find images of whiteboard notes
find screenshots with error messages
find outdoor photos with cars

This only works if the system can convert images and queries into comparable vectors.

image -> embedding vector
text query -> embedding vector
compare vectors -> ranked results

The goal is not that the model knows the correct label for every image. The goal is to return a useful candidate list faster than manual browsing.

That distinction matters. Semantic search is retrieval, not truth.

flowchart LR
  A[Folder images] --> B[Image encoder]
  B --> C[Image vectors]
  C --> D[(Vector index)]
  E[Query image or text] --> F[Query encoder]
  F --> G[Query vector]
  G --> H[Similarity search]
  D --> H
  H --> I[Ranked candidates]
  I --> J[Metadata filters and rerank]
  J --> K[Explainable results]

The model defines the meaning of “near”. A CLIP-style model can put text and images in a shared space. A visual-only model may be strong for image-to-image similarity but not support text queries directly.

Think of a map. A physical map gives coordinates for places:

Delhi -> latitude, longitude
Mumbai -> latitude, longitude

Nearby coordinates usually mean nearby places.

An embedding model gives coordinates for meaning:

beach image -> vector
forest image -> vector
invoice image -> vector
receipt image -> vector

In a useful embedding space:

beach images are near ocean photos
receipts are near invoices and document photos
screenshots are near other screenshots
whiteboard notes are near diagrams and handwriting

The vector space is learned from training data, so it is not perfect. But it gives the search system a geometry that filenames do not provide.

That geometry is what makes semantic search possible.

For an image, the conversion path is:

flowchart LR
  A[Image file] --> B[Decode image]
  B --> C[Resize/crop/normalize pixels]
  C --> D[Vision model]
  D --> E[Feature vector]
  E --> F[Normalize vector]
  F --> G[Store embedding]

The code shape is:

for image in images:
    pixels = decode(image)
    input_tensor = preprocess(pixels)
    vector = model(input_tensor)
    vector = normalize(vector)
    store(image_id, vector, model_name, dimension)

The preprocessing step matters. Most embedding models expect a specific input shape and pixel normalization policy:

resize to expected size
center crop or pad
convert to RGB
scale pixel values
normalize by model-specific mean/std

If preprocessing is wrong, the embedding can be bad even when the model is good. A model trained on normalized RGB inputs may produce poor vectors if it receives BGR pixels, unscaled values, or stretched aspect ratios.

Embeddings work because the model was trained to place related things near each other.

Different model families learn that geometry in different ways.

image -> model -> label probability

edges
textures
shapes
parts
object-level patterns
scene-level patterns

images that need similar features for classification often have similar internal representations

visual similarity
object/category grouping
basic image organization

not naturally aligned with text queries
may focus on classification categories more than general meaning

matching image and caption should have high similarity
non-matching pairs should have lower similarity

image: dog running on grass
text: "a dog running on grass"

image meaning and language meaning are trained into a shared space

text-to-image search
image-to-image search
natural-language visual queries
zero-shot categories

depends heavily on training data
can miss fine-grained visual details
can reflect caption biases

the model learns visual structure that remains stable under transformations

visual grouping
near-scene similarity
image clustering
fine visual structure

does not automatically support text queries unless paired with a text model or alignment step

This is the part many products hide.

Two models can produce different nearest neighbors for the same image.

A CLIP-style model may group by caption-like concept:

dog
receipt
screenshot
whiteboard

A self-supervised visual model may group by visual structure:

texture
layout
shape
scene composition

A perceptual hash may group by low-level visual fingerprint:

same image resized
same screenshot compressed
same photo lightly edited

So the product should not say:

these are objectively similar

It should say:

these are similar under this model and this scoring method

That is why the index should store:

model name
model version
embedding dimension
preprocessing config
normalization policy
created time
source file fingerprint

Many retrieval systems normalize embeddings before search:

v_normalized = v / ||v||

Why?

Because then dot product becomes cosine similarity:

dot(normalize(a), normalize(b)) = cosine(a, b)

Without normalization:

longer vectors can score higher because of magnitude

With normalization:

direction matters more than length

For semantic search, direction is usually the intended signal.

Before trusting search results, test the embedding model with small probes.

For image search:

query: one receipt image
expected: other receipts/invoices near top

query: one screenshot
expected: other screenshots near top

query: one outdoor photo
expected: outdoor photos near top

For text-to-image search:

query: "whiteboard notes"
expected: whiteboards, diagrams, handwritten boards

query: "terminal error"
expected: terminal screenshots or error dialogs

Quality checks should include bad cases:

similar colors but different meaning
same object but different context
documents with similar layout but different content
screenshots from different apps

The point is not to prove the model is perfect. The point is to learn what kind of similarity it provides.

Flat search compares the query vector against every indexed image vector.

best = []
for image in indexed_images:
  score = dot(query_vector, image.vector)
  if score >= min_score:
    best.add(image, score)
return top_k(best)

Visual model:

query vector
     |
     v
compare with image 1
compare with image 2
compare with image 3
...
compare with image N
sort by score

Flat search is exact and easy to debug. It becomes expensive when N grows large.

Flat search has complexity:

O(N * D)

Where:

N = number of images
D = embedding dimensions

For 10,000 images and 512-dimensional embeddings:

10,000 * 512 = 5,120,000 multiply-add operations

That is often fine on a desktop for moderate folders. Approximate search is an optimization after this exact baseline exists.

Image query search starts with an example image.

query image -> query embedding -> nearest image embeddings

Good uses:

• find visually similar shots;
• find near duplicates after editing or resizing;
• find images from the same scene;
• find related screenshots or documents.

Failure modes:

• the query model differs from the indexed model;
• query embedding dimension does not match the stored dimension;
• the example is visually ambiguous;
• metadata filters exclude the right answer;
• the top score is low but still returned because top_k forced output.

A good product checks embedding dimension and model identity. If the folder was indexed with one embedding model and the query uses another, scores are not meaningful.

Approximate nearest neighbor search avoids comparing against every vector.

It trades exactness for speed. The product question is:

Can we return nearly the same top results faster?

Not:

Can we change what similarity means?

The similarity definition stays the same. The index just narrows candidates faster.

query -> enter graph -> move to closer neighbor -> move again -> collect candidates

far node --- mid node --- close node --- nearest node
                \             \
                 \             other candidate
                  skip many unrelated points

flowchart TD
  A[All vectors] --> B[Train coarse centroids]
  B --> C[Assign vectors to buckets]
  D[Query] --> E[Find closest buckets]
  E --> F[Search only probed buckets]
  F --> G[Return approximate candidates]

Every embedding index can become stale.

A file can be:

modified
deleted
moved
renamed
replaced

A model can change:

old embeddings: model A, 512 dimensions
new query: model B, 768 dimensions

That comparison is invalid.

Each indexed record should store:

file id or path
file size
modified time
embedding model id
embedding dimension
embedding vector
index version

Before search:

if file changed:
    recompute embedding

if model id mismatch:
    require reindex

if dimension mismatch:
    reject query

This is not optional. A stale semantic index creates confusing failures because the UI looks correct while the data is wrong.

Minimal version:

for image in folder:
    embedding = embed_image(image)
    embedding = normalize(embedding)
    save(image_id, embedding, model_id, dimension, mtime, size)

query = embed_query(query_input)
query = normalize(query)

results = []
for record in records:
    if record.model_id != query.model_id:
        skip_or_reindex()
    if metadata_filter_fails(record):
        continue
    score = dot(query, record.embedding)
    results.append((record, score))

return top_k(results)

Production version:

add batch embedding
add cache invalidation
add query-time filters
add result explanations
add ANN index for scale
add reranking
add debug view
add rebuild controls

Used: image embeddings, normalized cosine scoring, SQLite-backed records, metadata filters, image-query search, text-query support where model-compatible, optional FAISS candidate narrowing, duplicate hash prefilters, and optional ORB rerank.

Removed or avoided: treating embeddings as exact labels, hiding model/dimension mismatches, and presenting approximate search as exact unless the path is flat.

Deferred: automatic model selection, universal text search for every embedding backend, and claims about semantic quality without a public benchmark.