ADR 0108: Multi-representant person representation: per-mode centroids with per-representative pessimistic matching (semisupervised-v2)
Status: accepted Date: 2026-10-09
Context
The single mean centroid per person under-represents persons whose faces form several visual modes (light / age / angle clusters): on the owner library (Okt 2026) Christian holds 1142 accepted faces of which 394 sit beyond 0.50 cosine distance from their OWN centroid, and the baseline single-centroid cross-validation measured top-1 86.3% / wrong-suggest 8.7% at the production band 0.50 — with Christian-to-Nathalie (51x) and Christian-to-Jesper (40x) as the dominant wrong suggestions. A k-fold cross-validation of multi-representant alternatives (ADR 0101 follow-up, PR #1080) decomposed each person's train faces by agglomerative average linkage (cosine distance, deterministic UPGMA with Lance-Williams updates and row-major argmin tie break) cut at a distance threshold, represented each cluster by its re-normalized mean centroid, and matched arriving groups by minimum distance. At threshold 0.50 the pessimistic rule (ADR 0106 margin computed PER REPRESENTATIVE with the representative's own cluster size as n_target) measured top-1 97.0% / wrong-suggest 2.7%; the Christian confusions drop to <3. The threshold sweep measured a better top-1 at 0.40 (2747/2796) but with ~7x the representatives (Christian ~218) — 0.50 is the shipped knee. Same suggestion volume as today is reachable by re-calibrating the suggest band from 0.50 to ~0.45 after deploy (WS 1.6% at equal volume; multi-rep distances shrink systematically: correct top-1 median 0.325 vs 0.409), a deployment decision that stays with the operator.
Decision
semisupervised-v2 (and only v2) represents each person by MULTIPLE representatives: per model pair, the person's assigned usable faces are decomposed by the shared deterministic average-linkage core (unit_rows, cosine_distance_matrix, average_linkage_merges, cut_dendrogram — moved verbatim into memories_people_worker.clustering, imported by the CV experiment so production is the measured implementation by construction) cut at the owner setting representative_split_distance (V155, NUMERIC(4,3), finite [0,2], people default 0.50; 2.0 = everything in one cluster = exactly the pre-0108 single centroid, the operator off-switch). Each cluster is represented by its unit mean centroid (CV-measured winner over the medoid variant); singletons are their own representative; a faceless person has none. The B outcome scores every whole-group identity per representative: d_pess = d + Zsigmasqrt(1/n_source + 1/size_rep) with each representative's OWN cluster size as n_target; the person's pessimistic key is the min over its representatives, the payload distance is the RAW distance of the pessimistically chosen representative, and score/targetUncertainty carry that key (ADR 0106 semantics, cluster-exact). distance stays raw and confidence keeps clamp(1 - distance) — the envelope contract is untouched and the contract version does not bump. The shortlist for isolation/ambiguity (ADR 0103/0105) keeps its raw formulas on the raw-min shortlist (nearest-mode geometry, NOT pessimistically corrected). Determinism: faces are clustered in ascending faceAnnotationId order; the decomposition is recomputed per run, nothing is persisted. v1 and the legacy methods keep the single-centroid representation byte-identically; qualityScore/isolation/ambiguity formulas, emission gates, review-only semantics and all web code are unchanged.
Consequences
Recognition quality: top-1 86.3% -> 97.0% and wrong-suggest 8.7% -> 2.7% at today's band in CV (validated on the owner library only, k-fold over 2796 faces / 36 persons — not a guarantee for other libraries). Cost: K representatives instead of 1 per person — more distance computations in the B outcome (household scale trivial: Christian 97 representatives x 512 floats at 0.50) plus an O(n^2)-memory / O(n^3)-time per-person clustering per run; persons above REPRESENTATIVE_SPLIT_MAX_FACES (2000) usable faces skip the split and keep the single centroid (pre-0108 behavior) — documented degradation, no job failure. Multi-rep distances are systematically smaller, so with the band unchanged more faces fall inside the suggest band; the operator decides whether to tighten suggest_max_distance to ~0.45 after deploy (measured: equal suggestion volume at WS 1.6%). The enqueue payload gains one additive v2-only settings key (representativeSplitDistance); older workers ignore it. The split threshold is a deployment-independent owner setting; a future threshold change needs no migration, only the next run recomputes. Known CV remainder: under multi-rep matching Christian becomes a confusion HUB in the other direction (his many modes match other people's faces up to 8x as wrong top-1) — monitored, revisit with band recalibration.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference