001Notes

Face Clustering From Pixels To People

On this page47

A from-scratch visual tutorial on face detection, landmarks, alignment, embeddings, thresholds, clustering, prototypes, and human review, backed by an actual ClusterLens run.

2 Sept 2026Project Lessons48 min readhigh confidence

Article details

Status: Building Publicly
Subcategory: ClusterLens
Last reviewed: 2 Sept 2026
Prerequisites: No computer-vision, machine-learning, or clustering knowledge required

47 sections

A photo application receives a folder containing holiday pictures, school photographs, screenshots, and landscapes. The request sounds harmless:

Put photographs of the same person together.

A person does this almost without noticing the intermediate steps. A computer must answer a chain of separate questions: Is there a face? Where is it? Is it large and clear enough? How should a tilted face be straightened? How can it become numbers? What does same person mean numerically? When is a score strong enough? What happens when the algorithm is uncertain?

This chapter constructs that chain from scratch. It uses ClusterLens as the working system, but it is a tutorial about the ideas—not a feature tour. Every panel is marked as a conceptual diagram, a controlled dry run, or an actual run. The actual run used 12 generated, non-personal portraits representing three fictional identities. The images were stripped of metadata, processed locally through ClusterLens, and never sent to a remote face service.

ClusterLens’s current production Faces workspace is human-only. Its production defaults may use SCRFD and ArcFace. The portable evidence profile here deliberately uses the official OpenCV CPU pair—YuNet and SFace—so the experiment can be repeated without presenting one model as universal truth.

1. Five face problems that are often confused

“Face recognition” is too vague to describe a system. Consider one query portrait and a folder of photos:

Task	Question	Output
Detection	Is there a face, and where?	rectangle, confidence, landmarks
Verification	Are these two faces the same identity?	similarity plus a threshold decision
Search	Which indexed faces resemble this query?	ranked candidates
Clustering	Which faces seem to belong together without names?	unlabeled groups and outliers
Identification	Which known identity best matches this face?	a proposed name or unknown

Clustering does not discover a person’s name. It creates anonymous groups such as cluster 0 and cluster 1. A human may later say “cluster 1 is Maya.” Identification is a separate and more consequential decision.

The first engineering lesson is therefore:

Never choose a model or metric until the question has been made precise.

The gallery changes the meaning of the task

Verification is usually called one-to-one (1:1): one probe is compared with one claimed identity. Identification and search are one-to-many (1:N): one probe is compared with an entire gallery. That difference is not administrative. If a single unrelated comparison has a small chance of scoring too highly, performing a million comparisons gives that rare event many opportunities to happen.

There is also a distinction between closed-set and open-set operation. Closed-set identification assumes the person is somewhere in the gallery and asks which entry is best. Open-set identification permits the answer “unknown.” A personal photo organizer needs the open-set mindset because every new person should not be forced into the closest existing name.

Clustering is different again. It has neither a claimed identity nor necessarily a known gallery. Its job is to propose structure that a person can inspect.

2. A photograph begins as pixels, not a person

When a computer decodes an RGB photograph, it obtains a rectangular tensor. A 1024×1024 photograph has the shape:

Decoded RGB image

X∈ℝ^{1024 × 1024 × 3}

Each location contains red, green, and blue values. None is explicitly labelled eye, nose, face, or identity.

Changing one pixel changes three numbers. Moving the same face 50 pixels right moves thousands of values to different tensor positions. Cropping, illumination, camera response, makeup, age, pose, and expression all change pixels even when identity does not change.

Generated portrait used to explain pixels — A portrait as a person sees it

One sampled pixelRGB(—)

Nearby coloured squares are numbers, not semantic labels. A detector must learn recurring spatial patterns from many examples.

A full-image embedding would also encode clothes, background, objects, and composition. Face clustering therefore starts by isolating the face region.

How does a detector learn visual structure from numbers?

A convolutional neural network slides a small learned filter across the image. At each location it multiplies nearby pixels by filter weights and adds the results. One filter may respond strongly to a vertical light-to-dark transition; later layers combine many earlier responses into curves, eye-like arrangements, and eventually face-like configurations.

For a single-channel patch X and a 3×3 filter K, one output cell is:

One convolution location

Yᵢⱼ=Σᵤ Σᵥ Kᵤᵥ Xᵢ₊ᵤ,ⱼ₊ᵥ + b

The same filter weights are reused at every location. That weight sharing lets one learned pattern be detected anywhere in the image.

5×5 brightness patch

3×3 vertical-edge filter

3×3 feature map

This is the first important form of invariance. The filter can respond to an eye edge at the left or right because it scans the whole feature map. Pooling, deeper receptive fields, varied training examples, and alignment add other tolerances. Invariance is learned and engineered; it is never unlimited.

3. Detection: finding a face before describing it

A face detector scans an image and proposes records of the form:

One detection

d=(x₁, y₁, x₂, y₂, c, L)

The rectangle runs from top-left (x₁,y₁) to bottom-right (x₂,y₂), c is detector confidence, and L contains landmarks.

The confidence answers “how face-like did the detector find this region?” It is not a probability that the person is Maya, not an image-quality score, and not a similarity score.

Portrait

Bounding box

Image size

Landmarks5 measured points

Detection can fail in two directions. A false negative misses a real face; that face can never reach clustering. A false positive treats a poster, statue, or texture as a face; downstream embeddings then describe nonsense. Lowering the detector threshold usually recovers more difficult faces while admitting more false detections. There is no free recall.

How face detectors evolved

It helps to view modern detectors as answers to the same question: where should expensive computation be spent?

Main representation

Search strategy

Outputs

Viola–Jones: reject empty windows cheaply

The 2001 Viola–Jones detector compared rectangular light and dark regions using Haar-like features. An integral image makes any rectangle sum available with four array lookups:

Integral image

IΣ(x,y)=Σᵢ≤x Σⱼ≤y I(i,j)

Once cumulative sums exist, a large rectangle is no more expensive to sum than a small one.

AdaBoost selected useful weak features from a huge candidate pool. A cascade rejected obvious background with early, cheap stages and evaluated harder windows with later stages. It was a landmark real-time design, but rigid rectangular contrast patterns and sliding windows struggle with pose, occlusion, and uncontrolled conditions.

HOG: count edge directions

Histogram of oriented gradients replaces raw intensity templates with local edge direction histograms. A face window produces horizontal and vertical derivatives (gₓ,gᵧ), gradient magnitude √(gₓ²+gᵧ²), and orientation atan2(gᵧ,gₓ). Nearby pixels vote into angle bins; contrast-normalized blocks form a descriptor for a classifier. HOG is more tolerant of illumination than raw pixels, but still relies on handcrafted gradients and a multi-scale sliding-window search.

MTCNN uses three CNNs in a cascade. P-Net rapidly proposes candidate windows across an image pyramid. R-Net rejects and refines them. O-Net performs the final box regression and predicts five landmarks. The name multi-task refers to learning classification, box regression, and landmark localization together. The cascade saves computation, but sequential stages and repeated pyramid processing can be awkward on modern accelerators.

RetinaFace: dense multi-scale predictions

RetinaFace is a single-stage, multi-level detector. A feature pyramid exposes high-resolution maps for small faces and semantically stronger low-resolution maps for large faces. Detection heads predict face/background scores, box offsets, and landmarks densely. Its research formulation also used extra supervision such as projected 3D face vertices. “Single-stage” does not mean one neural layer; it means proposals do not pass through a separate second detector network.

YuNet: tiny CPU-friendly detection

YuNet is designed for a strong speed/accuracy trade-off in a small model. The official OpenCV Zoo model emits a face box, five landmarks, and a score and is integrated through FaceDetectorYN. The May 2026 OpenCV Zoo export used in this article supports dynamic input dimensions. YuNet is the evidence detector, not ClusterLens’s production default.

SCRFD: redistribute samples and computation

SCRFD begins with an engineering observation: adding computation everywhere is wasteful. Its search procedure redistributes capacity among backbone, neck, and detection heads, while sample redistribution emphasizes scales that need training support—especially small faces. ClusterLens may select scrfd_10g_kps as its production human-face detector. The suffix indicates a compute family and keypoint output; it is not an identity model.

The progression is not “old algorithm bad, new algorithm good.” A tiny CPU app, a crowded photo archive, and a GPU video service occupy different accuracy, latency, memory, and deployment constraints.

4. Bounding boxes, area, clipping, and why tiny faces are hard

For a box (x₁,y₁,x₂,y₂):

Bounding-box geometry

w = x₂ − x₁;h = y₂ − y₁;A = w h

A box touching the image edge may be clipped; a tiny box contains little identity information after resizing.

Suppose a detected face is only 20×20 pixels. Resizing it to 112×112 does not invent eyelashes or skin texture; it stretches 400 spatial samples across 12,544 positions. A large tensor is not automatically an informative tensor.

Face width Edge clipping

🙂

Feature pyramids: let different resolutions specialize

A face occupying 16 pixels and one occupying 300 pixels should not be interpreted only on the same downsampled map. A feature pyramid carries information across resolutions. Fine maps retain positions for tiny faces; coarse maps see broader context with larger effective receptive fields.

Face size in source image

P3 · fine map

P4 · medium map

P5 · coarse map

Some detectors start with anchors: predefined boxes of several sizes and aspect ratios at each feature-map location. The network predicts whether an anchor contains a face and regresses offsets that move it toward the true box. Anchor-free designs predict centers, distances, or corners more directly. Either way, the raw output is a crowded set of hypotheses, not a neat final rectangle.

🙂

When detectors propose overlapping boxes, non-maximum suppression often compares them using intersection over union:

Intersection over union

IoU(A,B)=area(A ∩ B)area(A ∪ B)

IoU is 0 for disjoint boxes and 1 for identical boxes. Strongly overlapping proposals can be reduced to the more confident one.

Move box B

Intersection over union: .

Non-maximum suppression sorts boxes by confidence, keeps the strongest, removes boxes whose IoU with it exceeds a threshold, and repeats. Soft-NMS reduces scores rather than deleting boxes abruptly. Very aggressive suppression can erase two genuinely distinct faces in a crowd; weak suppression can produce duplicate face records.

5. Five landmarks: a small geometric description

ClusterLens stores five points from the detector: left eye, right eye, nose tip, left mouth corner, and right mouth corner. Their pixel coordinates might be:

left eye      (417.7, 402.4)     right eye   (605.4, 400.5)
nose          (516.5, 519.3)
left mouth    (429.1, 616.7)     right mouth (588.8, 617.0)

These points do not encode identity. They establish a coordinate frame. If the eyes slope upward, the head is rolled. If the inter-eye distance is small, the face is small or distant. If landmarks are implausible, alignment may be unreliable.

Detected face with five actual landmarks — Detector coordinates in the original image

Select detections in the previous panel; this ledger follows the same evidence record.

Five landmarks are a compact compromise. Dense alignment models may predict 68, 98, 106, 468, or more points, or directly estimate a 3D mesh. Dense landmarks better describe expression and non-rigid shape, while five stable points are often sufficient to normalize a recognition crop. More landmarks do not guarantee better identity embeddings: noisy points can introduce a more elaborate wrong warp.

The points also expose pose. The eye-line angle estimates in-plane roll. Relative nose displacement hints at yaw. Large yaw hides part of the face, which a 2D affine transform cannot reconstruct. Modern pipelines may use pose-aware models, 3D alignment, multi-view templates, or reject an unrecognizable crop rather than hallucinating missing evidence.

6. Alignment: give the embedder the same coordinate system

Imagine comparing passport photographs when one is upright, one is rotated 15 degrees, and one puts the face in a corner. A model could learn to ignore every possible transformation, but normalization can remove much of that nuisance first.

Alignment chooses a canonical five-point template in a 112×112 output and finds an affine transform that maps detected landmarks toward it:

Two-dimensional affine transform

a b t_xc d t_yxy1=x′y′

The coefficients can rotate, scale, shear, and translate. Pixels are then resampled into the canonical crop.

Face

Raw crop vs aligned embedding

The score measures how much the representation changed. It is not an alignment-quality probability.

🙂

→

The transform is estimated from corresponding points, then inverse mapping asks where each output pixel should sample the source. Bilinear interpolation blends four neighbouring source pixels when the requested coordinate falls between them. Interpolation is necessary, but it slightly changes texture; repeated warps compound blur. Alignment should therefore be reproducible and performed once from the source image.

In this actual run, mean cosine similarity among different views of the same generated identity rose from before alignment to after alignment. That supports the narrow claim “alignment improved consistency here.” It does not prove that every dataset or model benefits equally.

7. The complete image tensor ledger

“Convert the face to an embedding” hides several shape changes. Walk one portrait through the actual pipeline:

JPEG bytesvariable length
decoded RGB1024×1024×3 uint8
detected cropbox-dependent H×W×3
aligned crop112×112×3
model tensor1×3×112×112 float32
raw model output1×128 float32
normalized embedding128 numbers, norm = 1

The batch dimension 1 means one face. Moving channels before height and width is a layout change from HWC to CHW. Converting to float and applying the model’s expected scaling changes numeric representation. The neural network—not resizing or normalization—creates the semantic representation.

Preprocessing is part of the model

Two ONNX files that both accept 112×112 input may still expect different colour order, numeric range, mean, and standard deviation. Common recipes include RGB versus BGR, [0,255] versus [0,1], and (pixel−127.5)/128. Supplying BGR to an RGB-trained model does not cause a helpful type error; it silently changes the distribution presented to every learned filter.

Suppose one channel value is 200. Scaling to [0,1] gives:

Two different preprocessing recipes

200 / 255=0.784while(200 − 127.5) / 128=0.566

Both values are valid floats. Only the training recipe tells us which one the network learned to interpret.

The ledger is therefore also a debugging contract. Record image orientation handling, decoder colour mode, interpolation method, crop template, tensor layout, data type, scaling, model checksum, and embedding dimension. “Same architecture” is insufficient for cache reuse if any of these change.

8. What a face embedding actually is

An embedding is a fixed-length list of learned measurements. For this SFace run:

Face encoder

f(aligned face)=e ∈ ℝ¹²⁸

The network compresses 37,632 RGB channel values into 128 coordinates useful for distinguishing identities in its training objective.

One actual prefix is shown below. Coordinate 1 is not “eye colour”; coordinate 2 is not “nose width.” Meaning is distributed across combinations of coordinates.

Face

128 dimensions, L2 norm . Only the first 12 coordinates are displayed.

The vector is a representation because it preserves some relationships while discarding others. Ideally, pose, crop, and lighting change the vector a little; identity changes it a lot. The actual evidence later shows that this ideal is only approximate.

The 128 dimensions cannot be faithfully displayed on a flat page. PCA can project them to two dimensions for inspection, but projection throws information away. Nearby dots suggest a pattern; they do not replace calculations in the original 128-dimensional space.

Colour is the known synthetic identity and is used only to evaluate the experiment. Hover or tap a dot to inspect it.

Choose a point

What happens inside the encoder?

The encoder is the part that turns an aligned crop into a vector. Calling it a “black box” is convenient, but not helpful. We can open the box far enough to understand the transformations without pretending that one neuron has a tidy English label.

Start with an aligned 112 × 112 × 3 tensor. It contains 37,632 channel values. The encoder repeatedly applies learned transformations and produces new tensors. A tensor at an intermediate layer still has spatial positions, but it usually has many more channels than the original RGB image:

112 × 112 × 3

height × width × colour channels

becomes, conceptually,

56 × 56 × 64

height × width × learned feature channels

Those 64 channels are not 64 new colours. Each channel is the response of a different learned detector. One may become useful for a diagonal contrast; another for a curved texture; another may activate only when several simpler patterns occur in a useful arrangement. The labels are our interpretation. Training stores numeric weights, not words such as eye or jaw.

The following animation is a shape ledger, not the exact internal graph of SFace or ArcFace. Use it to follow what kind of object exists at every stage.

Tensor now

One value can use

Main change

1. A convolution does not search for a whole face

A small convolutional filter slides across the tensor. At each location it multiplies nearby values by its weights, adds them, and usually adds a learned bias. The same weights are reused everywhere. This weight sharing is why a useful edge pattern can be detected on the left or right side of the crop.

For one output channel, a simplified operation is:

One convolution output

y_h,w,c=b_c + Σ_i,j,k W_i,j,k,c x_h+i,w+j,k

The filter uses a local patch across every input channel. Training learns W and b. The output location h,w and output channel c identify one resulting activation.

The earlier convolution dry run used a hand-written edge kernel so every number could be inspected. Inside a trained encoder the principle is similar, but the filters are learned jointly and there may be hundreds of input and output channels. A filter in layer 20 is not looking directly at RGB; it is combining features created by layer 19.

After convolution, an activation function such as ReLU or PReLU introduces a non-linearity. Without non-linearities, stacking many linear layers would still collapse into one large linear transformation and could not express the complex boundaries required for faces.

2. Downsampling trades position for context

Encoders normally reduce height and width while increasing channel count. For example, a 56 × 56 × 64 tensor might later become 28 × 28 × 128, then 14 × 14 × 256. The smaller grid costs less to process. More importantly, each deep activation can depend on a larger area of the original crop.

That original-image area is its receptive field. Click through this conceptual face grid. The highlighted square is the portion of the input that one activation could theoretically use after progressively deeper layers.

Network depth

Illustrated field

Possible evidence

The exact receptive field depends on kernel sizes, strides, dilation, and the network graph. A useful recurrence is:

Receptive-field growth

r_l=r_l−1 + (k_l − 1)j_l−1andj_l = j_l−1s_l

r is receptive-field size, k is kernel size, s is stride, and j is the spacing in input pixels between neighbouring activations. A theoretical field says what can influence a unit; actual learned influence is usually concentrated unevenly inside it.

This explains the rough hierarchy:

early features can describe tiny contrasts and edges;
middle features can combine local contours, textures, and repeated shapes;
deep features can relate distant regions such as eye spacing and lower-face structure.

It does not mean the encoder builds a clean checklist of facial measurements. Distributed features overlap, and the same channel may respond differently depending on context.

3. Residual blocks help a deep model remain trainable

ResNet-family encoders contain shortcut connections. Instead of forcing a block to replace its input completely, it can learn a residual change:

Residual block

output=input + F(input)

If the useful change F is small, the shortcut preserves an easier information path. It also gives gradients shorter routes during training.

MobileFaceNet-style models pursue a different resource balance, using compact convolutions and bottlenecks to reduce mobile or CPU cost. The engineering details differ, but both families progressively transform spatial feature maps before producing one descriptor.

Batch normalization, learned affine scales, and activation functions also appear throughout many backbones. Their stored training statistics and parameters are part of the model. Switching a network between training and inference behaviour incorrectly can therefore alter embeddings even when the weights file is unchanged.

4. Global pooling turns maps into a list

Near the end, suppose the network has a 7 × 7 × 512 tensor. It has 512 feature channels, and each channel has a 7×7 spatial response map. Global average pooling averages every channel separately:

Global average pooling for channel c

g_c=1H · WΣ_h=1^H Σ_w=1^W X_h,w,c

One H×W map becomes one number. Applying the same operation to C channels produces a C-value vector.

Run one toy channel below. The highlight visits each cell and the displayed mean accumulates. A real network performs this operation efficiently across all channels rather than running a visible loop.

No cells have been pooled yet.

Pooling discards the final map’s exact coordinates, but it does not mean all geometry was ignored. Before pooling, the deep channels were created from spatial relationships across the aligned crop. Alignment makes those relationships more consistent: the same grid region is more likely to contain comparable facial evidence in different images.

5. Projection and normalization finish the descriptor

A learned projection mixes the pooled channels into the requested embedding dimension. If g is the pooled feature vector:

Final projection

e=W_eg + b_e

This is still learned: changing one pooled channel can affect many embedding coordinates. Some architectures add normalization layers or a training head around this projection.

During training, the embedding is connected to a loss-producing head. During deployment, that training head is usually removed. The encoder retains the geometry the loss taught it, and ClusterLens compares or clusters the emitted vectors.

A Vision Transformer offers another backbone. It splits the aligned face into patches, projects each patch into a token, adds position information, and uses self-attention to mix evidence between tokens. A token near an eye can interact directly with a token near the mouth instead of waiting for many local convolutions to enlarge its receptive field.

CNNs bring a strong locality bias: nearby pixels are combined first and the same filter is reused across positions. Transformers bring flexible global interaction, but they still need data, an objective, and positional structure that preserve identity evidence. Patch augmentation must not destroy the very relationships recognition needs. TransFace is one research example adapting transformer training and hard-sample handling to face recognition rather than assuming a generic image-classification recipe will transfer unchanged.

An embedding dimension is a capacity choice, not a semantic count. More coordinates can represent finer variation but cost memory and comparison work and can retain nuisance information. Fewer coordinates compress more aggressively. Changing 128 to 512 does not guarantee four times the identity information; architecture, loss, data, and training dominate that simplistic interpretation.

9. How a model learns identity-shaped geometry

The encoder does not begin with an identity-aware map. Before training, two photographs of the same person can be far apart, while two similarly lit faces can be accidentally close. Its parameters are numbers initialized without the finished recognition behaviour.

Training changes those parameters so the output space becomes useful for an objective. For supervised face recognition, the training data commonly contains an identity label for each crop:

Maya / studio light
Maya / side light
Maya / tilted pose

Daniel / studio light
Daniel / padded crop

Serena / studio light
Serena / smiling

The labels say which examples belong to the same person. They do not specify the embedding coordinates. Nobody tells the model “Maya must be at [0.4, −0.2, ...].” The loss only says which relationships or class decisions are desirable. Gradient-based optimization discovers weights that reduce violations across many batches.

The training loop, one causal step at a time

For a mini-batch, the loop is:

Decode, align, and preprocess several labelled face crops.
Run the encoder forward to obtain an embedding for every crop.
Use a training head or pair/triplet rule to calculate a loss.
Backpropagation calculates how each parameter influenced that loss.
An optimizer nudges the parameters in a loss-reducing direction.
Repeat with new identities, poses, qualities, and hard examples.

The basic gradient-descent update is:

One parameter update

w_new=w_old − η∂L∂w

L is the batch loss, ∂L/∂w is the gradient describing local sensitivity, and η is the learning rate. A real optimizer updates millions of parameters and may use momentum or adaptive statistics.

Imagine one parameter increases an edge response. If increasing it raised the loss, its gradient is positive and subtracting the gradient lowers the parameter. If increasing it helped, the gradient is negative and the update raises it. Backpropagation applies the chain rule through projection, pooling, deep blocks, and early filters so each parameter receives credit or blame.

The animation below compresses thousands of real updates into one slider. The points are conceptual two-dimensional embeddings. Real training happens in a much higher-dimensional space, and real trajectories are noisy rather than perfectly smooth.

Training progress

Within-identity spread

Between-identity gap

Training state

Two pressures are being illustrated:

invariance: different views of Maya should stay close despite pose, illumination, expression, or crop variation;
selectivity: Maya and Daniel should remain separated even if they share age, lighting, camera, or other superficial patterns.

That combination creates identity-shaped geometry. The phrase does not mean the model learns identity as a philosophical fact. It means the training objective organizes embeddings so identity labels predict neighbourhoods on the training distribution.

This geometry can fail in several ways. The model can memorize training identities without generalizing, use background or camera shortcuts, separate demographic groups unevenly, collapse multiple identities together, or treat pose as more important than identity. Dataset construction, sampling, augmentation, alignment, backbone capacity, and the loss all affect what the space preserves.

Before deep embeddings: Eigenfaces and local texture

Eigenfaces applies PCA to aligned training faces. Subtract the mean face, find directions of greatest pixel variation, and express each face as weights along those basis images. It elegantly turns a large image into a short coordinate list, but its highest-variance directions can describe lighting, pose, or background rather than identity. Its lesson survives: recognition begins by choosing which variation to preserve.

Local binary patterns take a different route. Around each pixel, compare neighbours with the centre, encode brighter/darker outcomes as bits, and histogram those patterns across facial regions. LBPH is interpretable and cheap but cannot learn the rich invariances of modern deep encoders.

Classification training

The most familiar training formulation resembles ordinary image classification. If the training set contains 100,000 identities, attach a classifier with 100,000 outputs. For a Maya crop, the desired answer is the Maya training class. Cross-entropy penalizes probability assigned away from that class.

This does not mean the deployed product can recognize only those 100,000 people. The classifier is temporary scaffolding. Its weight vector for each training identity behaves like a learned class direction. To classify many views correctly, the encoder tends to place each view near its identity’s direction. After training, the fixed classifier is removed and the penultimate features are used as embeddings for people never present in the training set.

For logits z₁…zC, softmax converts scores into a training distribution and cross-entropy penalizes the correct class y:

Classification loss

pᵧ=eᶻʸΣⱼ eᶻʲ;L = −log pᵧ

The classifier weights can be interpreted as training identity directions. Deployment keeps the encoder and discards the fixed list of training names.

Consider a three-identity batch. The correct Maya logit is largest, so the classification is technically correct:

Maya      2.2
Daniel    1.9
Serena   −0.3

But softmax assigns Maya only about 0.549 probability, and the loss is about 0.600. Daniel is dangerously competitive. If training later produces [5.0, 1.5, −1.0], Maya receives about 0.968 probability and the loss falls to about 0.033.

The important intuition is:

Crossing a decision boundary is enough to be correct, but it may not be enough to create a reusable verification space with a safety gap.

Ordinary classification optimizes the training classes. Face verification later asks whether two previously unseen samples are the same person. Margin objectives were developed to make the geometry around those class directions more explicitly compact and separated.

FaceNet and triplet loss

A triplet contains an anchor a, another view of the same person p, and a different person n. The desired inequality is:

Triplet objective

‖a − p‖² + α<‖a − n‖²

The negative should be farther than the positive by margin α. Hard examples supply useful pressure; already-separated examples contribute little.

Margin α

APN

The max(0, …) matters. Once a negative is already farther away than required, that triplet has zero loss. It supplies no new teaching signal. With millions of possible triplets, randomly choosing them can produce mostly satisfied, uninformative examples.

Triplet mining chooses more useful combinations:

an already-separated negative is comfortably far away;
a semi-hard negative is farther than the positive but still violates the requested margin;
a hard negative is closer to the anchor than the true positive.

Semi-hard and hard examples teach the boundary, but the hardest sample in a dirty dataset may be mislabeled, badly detected, or another image of the same person under the wrong name. Blindly emphasizing it can teach an error. Batch composition and mining policy are therefore part of the algorithm, not mere data-loader details.

Triplet loss also reasons about selected samples rather than every identity direction at once. It established the powerful idea that distances themselves can be the training target, but large-scale angular classifiers often use data more efficiently because every chosen class centre can participate in a single classification update.

ArcFace and angular margin

Face systems usually compare directions using cosine similarity. ArcFace makes training speak that same geometric language. Normalize the embedding x and every classifier weight Wⱼ to unit length. Their dot product is then the cosine of the angle between them:

Normalized class logit

Ŵ_j · x̂=cos θ_j

A smaller angle means the sample points more directly toward class j. A scale s is applied because cosine values alone occupy a small numerical range for softmax.

Suppose a Maya sample is 35° from Maya’s class direction and 50° from Daniel’s. Ordinary normalized softmax sees cos 35° ≈ 0.819 versus cos 50° ≈ 0.643, so Maya wins. ArcFace asks a stricter training question: would Maya still win if its target angle were made worse by a fixed margin? With a 15° margin, the target becomes cos(35° + 15°) = cos 50°. The sample must move closer to Maya’s centre before it wins comfortably.

ArcFace therefore adds a margin to the target angle before softmax:

Simplified ArcFace target logit

s · cos(θ_y + m)

s controls logit scale; m is the additive angular margin for the correct class.

Angular margin

The margin exists only in the training target logit. At inference time, ClusterLens does not add 15° to one identity or subtract a bonus from another. It runs the trained encoder and compares the resulting embeddings normally. The stricter rule has already influenced the learned weights and geometry.

OpenCV’s SFace is another recognition model with its own training recipe. Model families can output different dimensions and differently calibrated score distributions. A threshold copied from ArcFace cannot be assumed safe for SFace.

Why there are so many margin losses

The short answer is that “make faces separable” hides several different problems. Each method changes a different part of the training signal or makes a different trade-off. They should not be remembered as a list of brand names.

The progression is easier to understand as a sequence of questions:

Softmax
  Is the correct training identity on the winning side?

Metric learning
  Are same-identity samples closer than different-identity samples?

Angular margins
  Does the winning identity have geometric clearance?

Robust objectives
  Should every low-loss, hard, or suspicious example push equally?

Quality-aware objectives
  Should a blurred crop receive the same geometric demand as a clean crop?

Curriculum and sampled classifiers
  How can hard examples and millions of classes be trained efficiently?

These questions overlap, but they are not identical.

Problem 1: ordinary softmax allows a narrow victory

After normalizing embeddings and class weights, an ordinary class logit can be written as:

Angular softmax logit

z_j=s cos θ_j

The sample wins class j when its angle to that class direction is sufficiently smaller than its angles to competing directions. The common scale s changes softmax sharpness, not the angle ordering.

A sample 49.9° from Maya and 50.1° from Daniel can be correctly classified as Maya. Yet that 0.2° clearance is fragile. A small change in pose, compression, or domain may reverse the decision. Margin methods make training demand more than only reaching the correct side.

Problem 2: where should the extra clearance be measured?

SphereFace, CosFace, and ArcFace all penalize the correct class, forcing the encoder to compensate by moving the sample closer to its correct direction. They place that penalty in different coordinate systems:

Three margin locations

SphereFace:s cos(mθ_y)CosFace:s(cos θ_y − m)ArcFace:s cos(θ_y + m)

Only the correct-class logit is modified. SphereFace multiplies the angle, CosFace subtracts in cosine space, and ArcFace adds in angle space. Their hyperparameter values are not numerically interchangeable.

Use the toy comparison below. Maya’s class direction is at 0°, Daniel’s is at 50°, and the sample’s angle to Maya is adjustable. The translucent “penalized” ray translates each target logit back into an effective angle so all four choices can be drawn on one picture. That effective ray is an explanatory device; CosFace itself subtracts a cosine value rather than rotating a vector.

Sample angle θ

Training formula

Target vs rival

Training decision

At θ = 35°, ordinary angular softmax gives Maya approximately 0.819 and Daniel approximately 0.643, so Maya wins. A toy CosFace margin of 0.20 reduces Maya’s target to 0.619, so this sample no longer passes. A toy ArcFace margin of 15° evaluates Maya at cos 50°, exactly on the rival angle. Training must change the encoder so the unpenalized Maya angle becomes smaller.

This is the core intuition:

The loss temporarily makes the correct answer harder during training so the unmodified embedding has more clearance during inference.

SphereFace: multiply the target angle

SphereFace’s multiplicative angular margin helped make hyperspherical geometry explicit. Multiplying θ makes the correct logit drop more as the target angle grows. Its angular transformation and optimization behaviour require careful handling, which motivated easier-to-control formulations.

CosFace: subtract in cosine space

CosFace subtracts a constant from the correct cosine. It is direct to express after feature and weight normalization. However, cosine is non-linear in angle: subtracting 0.20 does not correspond to the same number of degrees at every starting angle.

ArcFace: add in angle space

ArcFace adds a constant angular margin. Degrees or radians give an intuitive geodesic interpretation on the unit hypersphere. That interpretability is one reason ArcFace became a common reference point, but it does not make one fixed margin universally optimal for every dataset and image quality.

Why not choose an enormous margin?

A larger margin is a stricter exam, not free accuracy. If it is too large:

low-quality but correctly labelled faces may be impossible to place deeply inside their class region;
mislabeled samples can produce destructive gradients;
limited model capacity may not satisfy all class regions simultaneously;
optimization can become unstable or converge poorly;
the training distribution may become beautifully separated while the target deployment domain still fails.

The margin is therefore a calibrated training hyperparameter. It is not an inference threshold, and it is not a percentage of identity confidence.

Problem 3: not every gradient deserves equal trust

An ordinary objective may let unusual or noisy samples dominate because they produce large loss. Some hard samples are valuable—they reveal confusing identities. Others are detector failures, extreme blur, occlusion, or label mistakes. A robust loss asks not only “is this sample wrong?” but also “how should its gradient be weighted?”

SFace takes this optimization-oriented view. It uses sigmoid-shaped functions to rescale intra-class and inter-class gradients according to similarity. The goal is to provide useful pressure without allowing every extreme pair to push with the same force. The OpenCV Zoo SFace model used in this article is a MobileFaceNet instance trained with the SFace loss and emits 128 values.

This is why SFace does not fit neatly into the sentence “one more angular margin.” Its distinguishing idea is how optimization pressure is shaped.

Problem 4: image quality changes what is achievable

A sharp frontal crop and a tiny blurred profile do not contain equal identity evidence. A fixed rule can over-punish the poor crop, asking it to occupy the same tight region as an unambiguous sample.

MagFace connects the pre-normalization feature magnitude to recognizability. It encourages magnitude to carry a useful quality ordering and uses magnitude-aware geometry around class centres. This produces two signals from one forward pass:

embedding direction
    → identity comparison

raw embedding magnitude
    → learned recognizability signal

That is why normalizing a MagFace feature immediately and discarding its raw norm throws away information the training objective intentionally organized.

AdaFace also uses feature norm as a quality-related proxy, but uses it to adapt the margin and training emphasis. Its motivation is that low-quality examples should not all be forced by the same rule used for clean samples. The norm is still model- and domain-dependent; it is not a universal aesthetic score.

Problem 5: what counts as useful difficulty changes during training

Early in training, the embedding space is chaotic. Extremely hard negatives may be noise or may exceed what the current model can interpret. Later, those boundary cases become informative. CurricularFace changes emphasis over the course of training, broadly moving from easier examples toward harder ones as the model becomes more capable.

This resembles teaching arithmetic before adversarial exam questions. The curriculum changes when examples dominate the gradient, while ArcFace and CosFace primarily change where a target must lie.

The next lab compares these families by the question they answer. Its pressure bars are qualitative teaching marks—low, medium, and high—not values reported by the papers or coefficients that can be copied into training code.

identity A region

identity B region

sample

Question answered

What changes?

Beginner intuition

Main caution

The sample position and pressure bars summarize one motivation. They are not measured performance rankings between the methods.

The correct question is not “which loss name is newest?” It is:

What failure dominates my data?

insufficient angular clearance?
bad triplet sampling?
label noise?
poor-quality evidence?
hard-negative scheduling?
classifier scale?

These methods primarily describe training objectives and representation geometry. They are not interchangeable post-processing buttons in ClusterLens. Changing the loss means training or adopting a different encoder, then rebuilding embeddings and recalibrating every downstream threshold.

Training scale and Partial FC

Modern face training can involve millions of identities. A conventional final classifier needs one weight vector per identity, making its matrix and gradient communication expensive. Partial FC samples a subset of negative class centres for each update while retaining the positive class, reducing memory and distributed-training cost. It changes how large training is made feasible; it is not used during local inference.

10. Normalization and cosine similarity

L2 normalization divides a vector by its length:

Unit-length embedding

ê=e‖e‖₂where‖e‖₂ = √Σ eᵢ²

After normalization, every embedding lies on a unit hypersphere.

For unit vectors, cosine similarity and dot product are the same:

Cosine similarity

cos(a,b)=a · b‖a‖₂ ‖b‖₂=â · b̂

1 means the same direction, 0 means perpendicular, and −1 means opposite. It is a geometric score—not a probability of identity.

Angle

cos(°) = .

Two different people can still score highly because the model, evidence domain, pose, rendering style, or training distribution makes their representations similar. Conversely, a difficult view of the same person can score lower. Geometry is evidence, not identity truth.

For unit-normalized vectors, Euclidean distance contains the same ordering information as cosine:

Cosine–Euclidean relationship on the unit sphere

‖â − b̂‖²=2 − 2 cos(a,b)

If cosine is 0.98, squared Euclidean distance is 0.04. This equivalence disappears when vectors are not normalized.

Normalization removes magnitude. That is convenient when magnitude is nuisance, but quality-aware models such as MagFace intentionally put information in the raw norm. A pipeline must know whether to preserve raw magnitude for quality assessment before normalizing for comparison.

11. The pairwise similarity matrix: where clustering begins

For N faces, compare every pair to form an N×N matrix:

Similarity matrix

S = EEᵀ;Sᵢⱼ = eᵢ · eⱼ

Rows of E are normalized face embeddings. The diagonal is 1 because every face is identical to itself.

Actual twelve by twelve cosine similarity heatmap — Actual aligned-embedding similarities. Bright blocks along the diagonal indicate four views of each generated identity.

Left face Right face

The same-identity scores ranged from to . Different-identity scores ranged from to . Those ranges overlap. That fact is more educational than a perfect demo: a model can make useful clusters without providing a universal cutoff.

Decision threshold

same generated identitydifferent generated identities

The genuine distribution answers how stable the model is across views of one identity. The impostor distribution answers how separated different identities are. A useful model needs both compact genuine scores and separated impostor scores. Reporting only an average similarity hides the dangerous tails where decisions fail.

12. A threshold turns a score into a decision

Verification applies a threshold t:

Pair decision

same identityifcos(a,b) ≥ t

Raising t rejects more impostors but can reject more genuine pairs. Lowering t accepts more genuine pairs but can merge different identities.

Four outcomes matter:

Reality	Decision accepts	Decision rejects
Same identity	true accept	false reject
Different identity	false accept	true reject

Threshold

True accepts

False accepts

False rejects

True rejects

False-accept rate False-reject rate

On this tiny generated dataset, t = 0.98 balances the two error rates at 4 false accepts and 1 false reject. That is an experiment result, not a production recommendation. Twelve stylistically related generated portraits cannot calibrate a population threshold. Real deployment needs representative validation data, separate operating points for the actual model, and scrutiny across relevant conditions and demographic groups.

FAR is not “the chance this match is wrong”

False-accept rate is measured over different-identity comparisons. Precision, the fraction of accepted comparisons that are genuine, also depends on how frequently genuine comparisons occur. In a large search gallery, most comparisons are impostors, so even a tiny per-comparison false-accept rate can yield unwanted candidates.

Gallery size Per-comparison false match rate

impostor comparisons give an approximate probability of at least one false match of .

The independence assumption in this classroom calculation is a simplification, but the systems lesson survives: a threshold calibrated for 1:1 verification cannot be transferred blindly to 1:N search.

13. Quality gates protect the representation stage

Some inputs should not be trusted enough to cluster. ClusterLens evaluates detector confidence, face size and area, aspect ratio, edge clipping, sharpness, luminance, contrast, and landmark/alignment availability. It classifies crops as clean, review, or reject.

Sharpness can be estimated with variance of the Laplacian: edges create strong second-derivative responses; a blurred image produces weaker variation. Luminance catches very dark or bright crops. Contrast catches nearly flat crops. These are heuristics, not measurements of human worth or identity confidence.

Face

Detector Sharpness Luminance Contrast

All 12 controlled portraits passed as clean. That verifies the clean path; it does not prove the reject path on real blur, occlusion, or extreme pose. A good evidence report says what it did not establish.

Perceptual quality is not recognition utility

A photograph may look attractive yet be poor for recognition because the face is tiny. Another may look noisy to a person but retain stable identity evidence. Face image quality assessment for recognition asks how useful an image is for matching, not only how pleasant it looks.

Modern directions include learned quality estimators, feature-norm proxies, landmark-guided transformers, and explicit recognizability models. Recognizability asks whether any matcher could reasonably distinguish the crop, regardless of whether detection succeeded. Blur, occlusion, low resolution, extreme illumination, and profile pose can all produce an unrecognizable detected face.

Recognition utility

identity centre

A quality gate should usually produce review or reject, not secretly enhance the image and pretend detail was observed. Face restoration can improve appearance while altering identity-relevant texture; synthesized pixels require separate validation and provenance.

14. Storing derived face records locally

ClusterLens uses SQLite rather than requiring a remote vector database. One indexed face is conceptually a record containing:

Field	Why it exists
image path + face index	stable local reference to one face in a photo
bounding box + detector confidence	reproduce and inspect detection
embedding blob + dimension	compare representations
quality status and metrics	filter or review questionable crops
hidden/tiny flags	product-level visibility policy
file `mtime` + size	invalidate stale derived data

The primary key is (image_path, face_index) because one photograph can contain several faces. If a file is edited while retaining its name, modification time and file size help detect that its old boxes and embeddings are stale.

Storedmtime=100 · 1 face

→

An embedding is derived, biometric-related data. Local storage reduces network exposure but does not remove privacy obligations. Access control, retention, deletion, and clear user intent still matter.

Storage, indexes, and model versions are separate layers

SQLite stores durable records and metadata. A matrix or nearest-neighbour index accelerates comparisons. The detector and embedder create the derived values. Keeping these responsibilities separate means the database can answer “which model produced this embedding?” and rebuild only incompatible rows.

If an old embedder outputs 128 dimensions and a new one outputs 512, padding or truncating vectors does not put them in one meaningful space. Even two 512-dimensional models learn different coordinate systems. Their scores are undefined across model versions. A safe migration creates a new model namespace, re-embeds sources, validates thresholds again, and removes old derived rows only when rollback is no longer needed.

Model A

Model B

15. Clustering from scratch

Clustering receives normalized vectors but no names. It must divide them using geometry.

Cosine K-Means

Choose K, assign each point to its closest centroid, recompute centroids, and repeat. With normalized inputs and re-normalized centroids, high dot product approximates spherical/cosine assignment.

K-Means objective

minΣ_k Σ_i∈Cₖ‖xᵢ − μₖ‖²

The algorithm minimizes within-cluster squared distance. It must be told K and tends to prefer compact groups.

Initialization matters because the objective is not globally solved by ordinary Lloyd iterations. K-Means++ spreads initial centres probabilistically, reducing the chance that several seeds start in one identity. Each iteration has two explicit phases: assign every face to its nearest centre, then average assigned vectors to move each centre. Stop when assignments or centres stabilize. Multiple random initializations can expose unstable solutions.

For face data, K is usually unknown. Estimating it from photo count is unreliable because one person may appear 500 times and another once. K-Means is nevertheless useful when the product has a plausible group count, as a comparison baseline, or inside a larger procedure.

HDBSCAN

HDBSCAN looks for groups that remain dense across distance scales. Sparse points may become noise rather than being forced into a cluster. It is attractive when the number of identities is unknown, but small or uneven groups can be difficult.

The intuition begins with core distance: how far must we expand around a point to include its min_samples-th neighbour? A lonely point has a large core distance. Mutual-reachability distance between two points takes the maximum of their ordinary distance and their two core distances. This stretches sparse regions, making tenuous connections expensive. HDBSCAN builds a hierarchy from those distances, condenses it using minimum cluster size, and selects groups that persist across density scales.

Mutual-reachability distance

d_mr(a,b)=max(coreₖ(a), coreₖ(b), d(a,b))

Sparse points cannot form cheap bridges based on one ordinary pairwise distance that happens to be small.

min_cluster_size asks how small a stable identity group may be. min_samples controls how conservative density estimation is. Raising either can turn small genuine identities into noise; lowering them can admit weak structures.

Threshold graph

Create a node per face and connect sufficiently similar neighbours. Connected components become groups. This has an intuitive failure: one accidental bridge can join two otherwise separate identities—a form of chaining.

A practical graph does not need all N(N−1)/2 edges. It can retrieve k nearest neighbours per node and keep only edges exceeding a similarity threshold. Requiring mutual nearest-neighbour agreement can remove one-sided accidents. Community-detection methods can split a connected graph more subtly than connected components. Learned graph clustering goes further by predicting whether edges or subgraphs represent the same identity. Every refinement adds parameters and calibration burden.

16. The actual backend comparison—and a useful failure

The evidence run gave all three backends the same 12 normalized SFace vectors and asked for an isolate-outliers policy.

Inspect backend

K-Means and HDBSCAN each recovered three identity-shaped groups and isolated Daniel’s padded view, yielding adjusted Rand index 0.879. The graph backend collapsed 11 faces from all three identities into one component and isolated one face, yielding ARI 0.0.

Why? Different-identity similarities reached 0.983. In a threshold graph, a few high-scoring cross-identity edges can form bridges. Connectivity is transitive: if A connects to B and B connects to C, A and C share a component even if they are not directly convincing matches. The failure is not “graph algorithms are bad.” It says this graph construction and operating point did not separate this evidence domain.

Adjusted Rand index is used only because the synthetic identities are known during evaluation. In a real unlabeled folder, no oracle supplies those answers.

Silhouette score measures whether points are closer to their own assigned group than to the nearest other group. For a point with mean within-cluster distance a and nearest-other-cluster distance b:

Silhouette coefficient

s=b − amax(a,b)

Values near 1 indicate separation, around 0 indicate a boundary, and below 0 suggest the point may fit another cluster better.

Internal metrics can prefer geometrically tidy but semantically wrong partitions. External metrics such as adjusted Rand index, normalized mutual information, or BCubed precision/recall require known labels. A trustworthy experiment therefore combines metric tables with visual member review and deliberately difficult cases.

17. What should happen to outliers?

An outlier may be a blurred view, unusual pose, false detection, rare identity, or a point the backend cannot place confidently. ClusterLens exposes three policies:

Policy	Behaviour	Risk
Assign	attach outlier to nearest acceptable group	can contaminate a person’s group
Isolate	make a singleton group	creates clutter but preserves uncertainty
Keep	retain backend noise status	requires UI support for unresolved items

● ● ●Daniel group

●padded view

Uncertainty is information. A product that hides it may look tidier while becoming less correct.

Outliers should also be reconsidered after the index changes. A singleton today may gain three neighbouring photos tomorrow. Conversely, removing a contaminated prototype may make an earlier assignment uncertain. Persisting raw evidence separately from current grouping lets derived decisions be recomputed.

18. Identity prototypes: summarize several examples

After a human labels multiple faces as Maya, ClusterLens can build a prototype by averaging their embeddings and normalizing again:

Identity prototype

p=Σᵢ eᵢ‖Σᵢ eᵢ‖₂

A prototype represents the central direction of accepted examples. It is not a photograph and cannot express every variation of a person's appearance.

Why average? One reference may be unusually lit or posed. Multiple accepted examples can reduce idiosyncratic noise. But averaging a wrongly labelled face contaminates every future suggestion, so prototype membership must remain inspectable.

Prototype Suggestion threshold

average + normalize →

unit prototype

The evidence generator deliberately requested proposals broadly so the pending-label storage path could be exercised. The slider above applies your chosen threshold to the measured prototype scores. It makes the distinction between “the code created a pending record” and “a safe policy should accept it” visible.

One centroid may not describe a person

Appearance varies with age, camera, facial hair, glasses, pose, and illumination. A single average can sit between several modes. A richer identity template can retain multiple prototypes—for example frontal, profile, older, and newer—or compare against the best of several vetted exemplars.

Video adds repeated observations. Rather than index every nearly identical frame, track a face through time, quality-weight its frame embeddings, and form one template:

Quality-weighted template

p=Σᵢ qᵢ eᵢ‖Σᵢ qᵢ eᵢ‖₂

High-utility frames contribute more, while many blurred frames should not overwhelm one clear frame through quantity alone.

Tracking introduces its own identity-switch failure: if two people cross, one track may accidentally combine both. Temporal continuity is another signal, not immunity from review.

19. A suggestion is not an identity

Naming is human-managed state. ClusterLens can suggest a person’s label for a cluster only when enough members support it. For a multi-face cluster, support should not rest on one lucky match. A simplified support rule is:

Cluster support requirement

required=max(2, ⌈members / 2⌉)

A singleton needs one supporting match; larger groups require at least two and roughly half the members.

Cluster members Members supporting Maya

Required support: . Decision: .

A UI should say “Suggested: Maya, review required,” not “This is Maya.” Similarity systems should communicate the epistemic status of their output.

The unknown decision needs two checks

The highest prototype score may still be weak. It may also be only slightly above the second-highest score. An open-set suggestion can require both an absolute threshold and a separation margin:

Two-part open-set rule

s₁ ≥ tands₁ − s₂ ≥ δ

s₁ is the best prototype score and s₂ is the runner-up. The second condition catches ambiguous near-ties.

Best score Runner-up Required gap

This does not turn scores into calibrated probabilities. It creates an explicit conservative region for unknown or review.

20. Pending labels, rejection memory, and undo

Automatic propagation in ClusterLens creates pending assignments. A human can accept or reject them. Rejected suggestions are remembered so the same unwanted proposal is not immediately recreated. Accepted batches retain enough information for undo.

Suggested: Danielprototype cosine 0.979

pending→awaiting human decision

This state machine is as important as the model. Without pending state, correction memory, and undo, one model mistake can silently become training-like evidence for more mistakes.

Feedback systems can amplify errors. If an automatically accepted face enters a prototype, that contaminated prototype may generate more high-scoring suggestions, which then enlarge the contamination. Safer designs distinguish human-confirmed, model-suggested, and rejected evidence; prototypes can be built only from confirmed examples or give suggested examples much lower weight.

Audit information should answer: which model and threshold produced the proposal, which prototype members supported it, what score and runner-up existed, who accepted it, and which records an undo will change. This is ordinary product engineering applied to machine uncertainty.

21. Failure modes: diagnose the stage, not just the result

pixelsdetectalignembedclusterreview

A merged cluster is not automatically a “clustering bug.” If landmarks put one eye on an eyebrow and the other on hair, the clusterer only receives the resulting distorted embedding. If two identities collide in the model space, changing K may rearrange rather than repair the evidence. Keeping boxes, crops, quality reasons, scores, and backend metrics makes the causal chain inspectable.

A disciplined diagnosis moves forward through the pipeline:

Verify decoding and orientation.
Overlay detections and landmarks.
Inspect raw and aligned crops.
Confirm preprocessing and model namespace.
Compare genuine and impostor score distributions.
Inspect nearest neighbours before clusters.
Compare backends and outlier policies.
Audit prototype membership and label provenance.

Changing the final cluster algorithm before checking upstream evidence can mask one symptom while preserving the cause.

22. Privacy, fairness, and scope

Face embeddings are compact, but they remain biometric-derived representations. They should not be treated as anonymous because they are not JPEGs. A safe local workflow should minimize collection, explain purpose, restrict access, support deletion, and avoid reusing the index for a different purpose without consent.

Threshold performance can vary with capture conditions and demographic groups. Aggregate accuracy can conceal unequal false-accept or false-reject rates. The evidence in this chapter uses only three generated identities, so it cannot establish demographic performance or suitability for consequential identification.

For a personal photo organizer, the goal is assistance under human control—not surveillance, not inferring sensitive attributes, and not claiming certainty from similarity.

Presentation attacks and synthetic media are separate problems

A recognizer can find that a printed photograph resembles an enrolled face; that is exactly what appearance matching asks it to do. Presentation attack detection asks whether the sensor is observing a live, physically present person rather than a print, replay, mask, or injection. It may use texture, motion, depth, infrared, challenge-response, or trusted capture signals. A photo organizer processing an existing library does not have liveness evidence and should not imply it.

Deepfake detection, morph detection, face restoration, attribute estimation, emotion estimation, and identity recognition are also distinct tasks. Sharing a face crop does not make their outputs interchangeable, reliable, or appropriate to collect.

NIST’s ongoing Face Recognition Technology Evaluation demonstrates why aggregate claims are insufficient: error behaviour depends on the algorithm, image quality, operating threshold, demographic cohort, and whether the task is 1:1 or 1:N. The responsible response is measured subgroup and condition evaluation, not assuming one fairness number transfers across deployments.

23. What “state of the art” means in 2026

There is no single state-of-the-art face algorithm independent of task and constraint. A detector can lead a hard-face benchmark while being too large for a local CPU. A recognizer can perform strongly on controlled portraits yet degrade under surveillance resolution, masks, aging, twins, or domain shift. Results also depend on training data scale and curation, alignment, test protocol, and the false-match operating point.

The modern research frontier includes:

Direction	What it tries to improve	Why it is not a drop-in answer
Efficient multi-scale detectors	small/occluded faces per unit compute	hardware and crowd density change the trade-off
Quality-aware losses	reduce damage from unrecognizable or noisy samples	feature norm and margins are model-specific
Transformer backbones	richer global and patch interactions	require specialized augmentation, mining, and large data
Large-scale classifiers / Partial FC	train on millions of identities	solves training infrastructure, not local inference policy
Recognizability and FIQA	decide when not to compare	a quality model itself needs domain validation
Set/video templates	combine several observations	tracking switches and correlated frames can contaminate templates
Foundation-model studies	test whether general representations transfer	domain-specific face models often remain stronger and easier to calibrate
Harder evaluation sets	expose saturation on classic benchmarks	leaderboard order can change across capture conditions

Classic benchmarks can become saturated, making many methods appear almost identical. Newer “Goldilocks” evaluation work deliberately seeks test sets that are neither trivial nor impossibly hard and shows that method ordering can shift across difficult conditions. The useful lesson for ClusterLens is not to chase the newest acronym. It is to preserve a replaceable detector/embedder boundary, version every representation, and run the intended local workflow on representative photos.

For this project today, SCRFD/ArcFace remains a sensible production-oriented default path while YuNet/SFace provides a portable CPU evidence path. MagFace, AdaFace, transformer recognizers, and learned FIQA are valuable experiment candidates—but each would require a new reproducible evidence profile, threshold sweep, clustering comparison, resource measurement, and safety review.

24. Complete actual run: one face through the whole system

Choose one record and step through everything that actually happened:

Record

The run’s final numbers are not a victory banner:

12 / 12 faces detected and quality-gated cleanly
128-dimensional normalized embeddings
same-identity aligned cosine mean     0.991241
different-identity cosine mean        0.954315
best tiny-set threshold               0.98 (4 false accepts, 1 false reject)
K-Means / HDBSCAN                     3 groups + 1 outlier, ARI 0.879121
threshold graph                       1 mixed group + 1 outlier, ARI 0.0

The experiment shows four things at once:

Detection and alignment can make a useful, repeatable representation pipeline.
Same-identity views can be geometrically close.
Different identities can also be uncomfortably close under a particular model and synthetic domain.
Clustering behaviour depends on the algorithm’s definition of a group, not just the embeddings.

That is the transferable lesson from Face Clustering in ClusterLens:

A face-clustering system is not one recognizer. It is a sequence of lossy representations, thresholds, grouping assumptions, persistent state, and human decisions. Correctness comes from making every boundary visible—and making uncertainty reviewable.

Evidence card

Schema
Generated
Dataset: 12 metadata-free, generated portraits; 3 fictional identities × 4 controlled views
Models: Official OpenCV YuNet 2026 May detector and SFace 2021 December embedder, CPU
Environment: Python · OpenCV · ONNX Runtime
Scope limit: Educational systems experiment; not demographic validation, deployment calibration, or identity ground truth

Primary references

Face Clustering From Pixels To People

1. Five face problems that are often confused

The gallery changes the meaning of the task

2. A photograph begins as pixels, not a person

How does a detector learn visual structure from numbers?

3. Detection: finding a face before describing it

How face detectors evolved

Viola–Jones: reject empty windows cheaply

HOG: count edge directions

MTCNN: coarse proposals, then refinement

RetinaFace: dense multi-scale predictions

YuNet: tiny CPU-friendly detection

SCRFD: redistribute samples and computation

4. Bounding boxes, area, clipping, and why tiny faces are hard

Feature pyramids: let different resolutions specialize

5. Five landmarks: a small geometric description

6. Alignment: give the embedder the same coordinate system

7. The complete image tensor ledger

Preprocessing is part of the model

8. What a face embedding actually is

What happens inside the encoder?

1. A convolution does not search for a whole face

2. Downsampling trades position for context

3. Residual blocks help a deep model remain trainable

4. Global pooling turns maps into a list

5. Projection and normalization finish the descriptor

9. How a model learns identity-shaped geometry

The training loop, one causal step at a time

Before deep embeddings: Eigenfaces and local texture

Classification training

FaceNet and triplet loss

ArcFace and angular margin

Why there are so many margin losses

Problem 1: ordinary softmax allows a narrow victory

Problem 2: where should the extra clearance be measured?

SphereFace: multiply the target angle

CosFace: subtract in cosine space

ArcFace: add in angle space

Why not choose an enormous margin?

Problem 3: not every gradient deserves equal trust

Problem 4: image quality changes what is achievable

Problem 5: what counts as useful difficulty changes during training

Training scale and Partial FC

10. Normalization and cosine similarity

11. The pairwise similarity matrix: where clustering begins

12. A threshold turns a score into a decision

FAR is not “the chance this match is wrong”

13. Quality gates protect the representation stage

Perceptual quality is not recognition utility

14. Storing derived face records locally

Storage, indexes, and model versions are separate layers

15. Clustering from scratch

Cosine K-Means

HDBSCAN

Threshold graph

16. The actual backend comparison—and a useful failure

17. What should happen to outliers?

18. Identity prototypes: summarize several examples

One centroid may not describe a person

19. A suggestion is not an identity

The unknown decision needs two checks

20. Pending labels, rejection memory, and undo

21. Failure modes: diagnose the stage, not just the result

22. Privacy, fairness, and scope

Presentation attacks and synthetic media are separate problems

23. What “state of the art” means in 2026

24. Complete actual run: one face through the whole system

Evidence card