ClusterLens Algorithms: Clustering Image Embeddings From First Principles

In a hand-designed vector, coordinate one might mean “redness” and coordinate two might mean “edge density.” Learned embeddings rarely offer that clean interpretation. Training shapes many coordinates together so that useful relationships emerge from the whole direction.

This is similar to a city map. A location is meaningful because every place uses the same latitude/longitude coordinate system. A latitude from one map and an arbitrary first coordinate from another model cannot be compared just because both are numbers.

The backend returns labels by row number. If paths are reordered later while the label array stays fixed, the arithmetic can be perfect and the product can still show the wrong images.

A safe sequence is:

scan paths
→ freeze their order
→ build/load embeddings in that order
→ cluster rows
→ map labels back through the frozen order
→ apply display filters

Within the current clustering service:

Those names do not select an entirely different semantic or visual model. Model selection happens earlier. Here the modes compare a PCA-projected space with the full normalized space.

The service records this explicitly:

semantic projection | input 768 -> PCA 50 | normalized

cosine full vector | input 768 -> full 768 | normalized

For one cluster (C):

If A=(1.0,1.0), B=(1.2,0.8), and C=(0.8,1.3) belong together:

After centroids stop moving, every point belongs to the centroid with the smallest squared distance. In two dimensions, each centroid owns a region of the plane called a Voronoi cell. The boundary between two centroids contains points equally far from both:

That shape constraint explains several failures better than memorizing a list. A crescent cannot be represented by one convex nearest-centroid territory without also claiming some empty or foreign space inside the curve.

Switch among four exact teaching datasets. Point fill is the intended group; the outer ring is the final KMeans assignment. A point whose fill and ring disagree is counted as a mismatch.

“Teaching mismatch” is available because this synthetic dataset has an intended colour. A real unlabelled folder has no automatic answer key.

Calling every row “KMeans” would hide meaningful differences. The representative, distance, initialization, batch strategy, and constraints are all part of the contract.

1. Inertia: how tightly does KMeans fit the points?
2. Silhouette: are points closer to their own group than to a competitor?
3. Workflow: are the resulting groups manageable for a person to review?

Inertia always falls or stays equal as k grows because extra centroids add flexibility. Selecting the smallest inertia therefore selects the largest allowed k.

Try the actual sweep:

Useful product heuristics include:

This is a product decision, not a theorem about the natural number of concepts in the folder.

Three common shortcuts are wrong:

FAISS = vector database             ✗
FAISS = approximate search          ✗
FAISS = GPU-only                    ✗

FAISS indexes live inside an application unless that application builds the surrounding database behavior. FAISS supports exact and approximate indexes, and it has CPU as well as GPU implementations.

The distinction is easier to see as a stack:

product behavior
  file paths, deletion, permissions, metadata, persistence

FAISS library
  vector indexes, distance kernels, clustering, quantization

mathematical operation
  L2 distance, inner product, nearest neighbour, KMeans objective

Changing the middle layer can make an operation faster without changing the question at the bottom. Changing the metric or index family can change the question or make its answer approximate.

FAISS accepts several queries together. If there are Q queries and we request the nearest K neighbours for each, both outputs have shape:

Example:

3 query vectors × top 2 neighbours

indices =
[[ 0,  1],      ← query 0
 [17,  4],      ← query 1
 [ 8, 11]]      ← query 2

Batching is not only a convenient API shape. Native code can process matrix blocks, reuse caches, use vector instructions, and keep worker threads busy more effectively than repeated one-query calls.

Recall Lloyd’s KMeans loop:

assign every point to its nearest centroid
                     ↓
replace each centroid with its assigned mean
                     ↓
repeat

The assignment step is itself nearest-neighbour search:

database inside the temporary index = K centroids
queries sent to that index           = N image vectors
requested neighbours                 = 1
returned row ID                       = cluster label

This inversion is worth pausing over. During ClusterLens’s FAISS KMeans assignment, the index contains the centroids, not all image embeddings. Every image embedding is a query asking, “which one centroid is nearest?”

Use a four-point dry run:

points:       A=(1,1)  B=(1,2)  C=(8,8)  D=(9,8)
centroids:   μ₀=(1,1)            μ₁=(9,8)

nearest centroid search returns:
A → 0    B → 0    C → 1    D → 1

mean update:
μ₀ ← mean(A,B) = (1,1.5)
μ₁ ← mean(C,D) = (8.5,8)

The search(..., 1) result is the label array [0,0,1,1]. Another iteration uses the updated means.

FAISS’s own implementation notes identify assignment as the dominant KMeans cost. For N vectors, K centroids, D dimensions, and T iterations, a useful work estimate is:

Change the workload below, then play one FAISS-style iteration. The animation shows the execution phases; the counters show why dimension and cluster count matter even when the number of images stays fixed.

The experimental clustering helper is short:

d = embeddings.shape[1]
kmeans = faiss.Kmeans(d, num_clusters, niter=20, verbose=False)
kmeans.train(embeddings)
_, assignments = kmeans.index.search(embeddings, 1)
labels = assignments.reshape(-1)

Line by line:

1. d fixes the width of every vector accepted by this run.
2. Kmeans(...) configures num_clusters centroids and at most 20 iterations.
3. train(...) alternates assignment and centroid updates.
4. kmeans.index exposes the trained centroid index.
5. search(embeddings, 1) asks for one nearest centroid per image.
6. the N × 1 ID matrix is flattened into N integer labels.

The production helper does not pass an explicit FAISS seed or restart count; those use the installed library’s defaults. The controlled evidence runner does pass seed 42. This difference belongs in any reproducibility claim.

The input has already passed ClusterLens’s preparation contract, so it is a dense float32 matrix. In cosine mode those rows are unit-normalized. FAISS KMeans then uses its configured KMeans assignment behavior; “the inputs were normalized” and “the algorithm is spherical KMeans” are not interchangeable claims.

Select a model to compare the measured scikit-learn and FAISS KMeans runs on the same 500 image embeddings:

ARI in this lab compares each result with the ten known CIFAR labels. It does not directly compare the two partitions with one another. Cluster numbers are arbitrary, so comparing raw label 3 from one implementation with raw label 3 from another would also be invalid.

Suppose ClusterLens grows from a single-user desktop application into a service used by many people. The vector calculation does not suddenly change:

What changes is everything surrounding the calculation.

The service may need to answer:

• Which user owns this vector?
• Does the record still exist after a restart?
• Can a query restrict results to folder = cars and year >= 2024?
• What happens while another process updates or deletes the record?
• How are writes recovered after a crash?
• How do multiple machines share the collection?
• Who is authorized to read this tenant’s vectors?
• How are keyword and vector scores combined?

FAISS supplies useful primitives for several of these workflows. It can store IDs in suitable index wrappers, serialize indexes, perform some forms of ID selection, and power indexes inside larger systems. It does not by itself become the entire networked, transactional, metadata-aware product.

Think of the difference this way:

FAISS
    a high-performance search engine on a workbench

vector database
    the engine plus shelves, durable labels, receiving desk,
    inventory rules, filters, concurrent access, and operations

Suppose a user asks for three images similar to a sports-car query, but only inside Alice’s cars folder.

The global similarity order is:

If the application retrieves global top three and filters afterward, only one result survives. It cannot recover ranks four and five because it never asked for them.

Compare three strategies:

This is a conceptual comparison. Real databases combine scalar indexes, graph traversal, partitions, bitsets, iterative scans, or internal query planning in different ways. “Supports filters” does not reveal whether a strict filter keeps latency and recall healthy. That must be tested with realistic filter selectivity.

Qdrant is a vector database service organized around collections of points. A point has an integer or UUID identifier, one or more vectors, and optional JSON payload.

collection: photos

point:
  id       = photo-184
  vector   = [0.14, -0.07, ...]
  payload  = {owner: alice, folder: cars, year: 2025}

Clients communicate through HTTP, gRPC, or language SDKs. Qdrant stores points in segments; distributed collections can be split into shards.

Its usual dense-vector path uses HNSW. Payload indexes accelerate fields used for filtering, and Qdrant’s filterable HNSW adds graph connections intended to keep traversal useful when some nodes fail the payload condition.

Why choose it:

• retrieval frequently combines semantic similarity with structured filters;
• JSON payload fits the application’s record shape;
• a dedicated self-hosted or managed vector service is acceptable;
• dense, sparse, named, or multiple vectors per point are useful.

How it differs from FAISS: Qdrant owns network APIs, point persistence, payload indexes, segments, background optimization, sharding, and distributed service behavior. FAISS exposes lower-level algorithms that an application can compose without adopting Qdrant’s record or server model.

Weaviate stores JSON-like objects in typed collections. An object has properties and normally one or more vectors.

Within a shard, it combines different structures:

object store      → retrieve the stored object
inverted indexes  → property filters and BM25 keyword search
vector index      → semantic nearest neighbours

Its vector index can be HNSW, flat, dynamic, or another supported option. The dynamic choice begins flat and can switch to HNSW after a configured object threshold. Hybrid search runs vector and BM25 retrieval and fuses their scores.

Why choose it:

• the domain naturally looks like objects with searchable properties;
• keyword, semantic, hybrid, and filtered search belong in one service;
• integrated vectorization modules are useful, while supplying your own vectors remains possible;
• per-collection or multi-tenant vector indexes fit the application.

How it differs from FAISS: Weaviate couples the vector index with object storage and inverted indexes, exposes database APIs, and manages CRUD at the object level. FAISS leaves object storage, keyword indexes, schema, and service behavior to the embedding application.

This table describes ownership boundaries, not a performance leaderboard. Any row can win or lose on a particular dataset.

It does not decide:

• which embedding model represents the objects;
• whether a 512-dimensional vector is better than a 768-dimensional one;
• whether cosine, dot product, or L2 matches the product meaning;
• how source objects are split into chunks;
• whether a retrieved result is useful to a person;
• whether an approximate index has acceptable recall.

Some products can run an embedding model for you. That combines two stages in one API; it does not erase the representation decision. The model name, version, preprocessing, dimension, and update policy are still part of the data contract.

Imagine lowering water around a landscape. Mountain peaks appear first as small islands. As more land appears, islands grow and sometimes join. A useful cluster is like an island that persists across a substantial range of water levels rather than appearing for one fragile instant.

The analogy becomes more useful when animated. The left hill is dense and tall; the right hill is smaller; a few weak points form a bridge. Increase the density requirement to remove less persistent points.

This is a conceptual hierarchy, not a claim that HDBSCAN deletes points with one literal slider. The actual algorithm builds a mutual-reachability tree, condenses it, and evaluates branch persistence across levels. The animation shows why persistence contains more information than choosing one global radius.

If core(A)=0.7, core(B)=0.5, and d(A,B)=0.4:

min_cluster_size removes branches that never become large enough. HDBSCAN measures how long candidate groups persist across density levels and selects a non-overlapping set, using ClusterLens’s eom or excess-of-mass selection.

One common stability expression integrates cluster size over inverse-distance density:

ClusterLens currently requests HDBSCAN’s EOM selection. Changing to leaf selection would not be a cosmetic setting: it changes which level of the hierarchy becomes the displayed grouping.

Suppose the best and second-best centroid scores are 0.61 and 0.60. argmax still returns the first cluster, but its margin is only 0.01.

A safer policy might require:

best score ≥ 0.70
and
best score − second-best score ≥ 0.08

The correct numbers must be calibrated on representative data. The important beginner lesson is structural: choosing the maximum and deciding that the maximum is trustworthy are two separate operations.

Centroid ranking answers “which member is closest to the average direction?” That is useful for a cover thumbnail, but it can show four nearly identical images and hide the group’s range. Compare three orderings of one fixed six-image beach cluster:

A medoid is another useful term. It is the actual member whose average distance to all other members is smallest. A centroid may not correspond to any real image; a medoid always does. For normalized embeddings and compact groups, centroid-nearest and medoid choices are often similar, but they are not mathematically identical.

For review tools, a strong layout often combines several intentions:

1. show one centroid-near cover;
2. show a few diverse representatives so the range is visible;
3. occasionally sample random members to expose mistakes that centrality hides;
4. let the user sort by filename or time when chronology matters more than geometry.

With CLIP and the recorded settings, KMeans produced:

10 clusters
0% noise
ARI       0.715
purity    85.8%
silhouette 0.100

Those statements answer different questions:

• 10 clusters is partly a consequence of asking for k = 10.
• 0% noise follows from KMeans assigning every point.
• ARI 0.715 says the predicted same/different pair relations agreed fairly strongly with CIFAR-10 labels.
• 85.8% purity says most members followed each cluster’s majority label.
• 0.100 silhouette says the embedding clusters were not separated by enormous geometric gaps.

There is no contradiction between good label agreement and a modest silhouette. Real image classes overlap, and the embedding model was not trained specifically to reproduce CIFAR-10’s ten names as ten spherical islands.

For DINO, the recorded graph setting reported 100% purity—but labelled 98.2% of images as noise. The few surviving tiny components were pure. Almost the whole dataset was left ungrouped.

This is the clearest reason to put coverage beside purity.

The displayed clustering times exclude model loading and embedding extraction. They measure only the recorded backend call on 500 prepared vectors. At this scale, process noise and implementation constants matter more than an asymptotic slogan.

For realistic sizing, the separate synthetic runtime artifact uses fixed 64-dimensional blobs at 500, 2,000, and 10,000 rows. It is useful for observing growth on this machine, not for predicting another machine’s production latency.

For a new collection:

1. Freeze a versioned sample and, where possible, a small labelled evaluation slice.
2. Cache embeddings together with model ID, preprocessing signature, dimension, file size, and modification time.
3. Run an exact, simple baseline such as normalized KMeans.
4. Sweep the parameter that changes the backend’s meaning: k, density size, or graph threshold.
5. Record cluster count, noise, coverage, size distribution, silhouette, and stability.
6. When labels exist, also record ARI, NMI, and purity.
7. Render representative members and random members from every cluster.
8. Inspect errors, bridges, tiny groups, and outliers—not only the prettiest clusters.
9. Re-run with another seed and a shuffled row order.
10. Store raw-backend and post-processed results separately.

The whole system can be summarized as a series of explicit contracts:

IMAGE FILES
  │  path order + invalidation metadata
  ▼
EMBEDDING MODEL
  │  model ID + preprocessing + output dimension
  ▼
PREPARED MATRIX
  │  optional fitted PCA + normalization + row map
  ▼
RAW BACKEND
  │  requested backend + actual implementation + raw labels
  ▼
POST-PROCESSING
  │  outlier policy + tiny-group policy + membership changes
  ▼
PRESENTATION
  │  centroid ranking + explanations + review actions
  ▼
EVALUATION
     geometry + coverage + labels + stability + human inspection

Every arrow has state that must be preserved or measured. If the model changes, the vectors are stale. If PCA is refitted, old projected points are incompatible. If rows move, labels attach to the wrong paths. If post-processing changes membership, raw quality no longer describes the gallery.

Inspect those arrows as contracts. Selecting a layer shows what it consumes, what it must emit, and the most common silent failure at that boundary.