This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
ADR 0122: Real-face multi-medoid identity representatives for semisupervised-v6 (amends ADR 0119, relativizes ADR 0108's centroid choice)
Status: proposed Date: 2026-11-14
Context
ADR 0108 ships per-split-cluster representatives as unit mean centroids, justified by the sentence 'CV-measured winner over the medoid variant'. The recovered G1 measurement behind that sentence (old w600k-v1 space, 2796 faces / 36 persons) was: medoid rule top-1 98.28% / wrong-suggest 1.72% vs representative-centroid 98.26% / 1.74% — a 0.02 pp difference, well under one face in 2796: within noise. The centroid was shipped on a coin flip, not a measurement.
Two later facts make the choice matter:
- Synthetic centroids qualify faces no real face of the person reaches. Incident class (e3403b1b): a face's distance to every own-face embedding of the person is 0.392 > the 0.35 gate, but the centroid sits within 0.35 → an m=1 B card is minted against a point in embedding space that is nobody, and a shadow identity sits nearer → the card is almost certainly wrong. This is exactly the 'single-image add-to-existing suggestion lands on the wrong person' class the owner reports.
- Fresh shadow-aware CV (2026-11-14, k=5 seed=42, cache 12107 faces / 805 persons incl. 745 stashed shadow identities as distractors; space arcface-r50/w600k-v2-aligned 512d; band 0.5; rules per ADR 0108 split at 0.50 — 'multi' = one representative per split cluster):
| group size | rule | SINGLE medoid (threshold 2.0, 1 rep/person): top1 / wrong | MULTI medoid (split 0.50): top1 / wrong |
|---|---|---|---|
| m=1 | medoid | 70.83% / 5.07% | 94.88% / 3.15% |
| m=1 | medoid-pess | 74.68% / 4.80% | 94.68% / 2.92% |
| m=2 | medoid | 87.64% / 4.12% | 95.69% / 3.18% |
| m=3 | medoid-pess | 94.41% / 1.40% | 96.85% / 2.45% |
| m=5 | medoid-pess | 95.03% / 1.86% | 96.27% / 2.48% |
m=1 is the headline: 12040 evaluated single-face groups → single medoid 8528 top-1 / 610 wrong; multi medoid 11423 / 379 — +24.05 pp top-1 and −37.9% wrong-suggest on precisely the card class the owner complains about. The production-relevant comparison (today's centroid reps vs multi-medoid) was not re-run because the centroid variant's defect is structural (point 1): a centroid can be closer than every real face, which no threshold recalibration fixes.
ADR 0119 (merged to main, not yet enabled on any production domain) already redefines v6's representative selection (quality-gated face set) while keeping ADR 0108 centroids as the representative kind. ADR 0114 already computes, ships, and persists the per-cluster medoid face id (thumbnail rule: manual pin, else medoid of largest cluster) in the same decomposition pass — [(vector, clusterSize, medoidFaceAnnotationId)] per representative, cached via the ADR 0113/V168 fingerprint keyed on the selected face-id set + split distance + model pair. The medoid machinery exists in production for display; this ADR promotes it to identity evidence.
Decision
V1 — Representative vector = the split cluster's medoid face, not its centroid (v6 only). For semisupervised-v6, each ADR 0108 split cluster is represented by the unit-normalized embedding of its medoid member: the member minimizing the sum of cosine distances to the other members of its cluster, ties broken by smallest faceAnnotationId — verbatim the existing ADR 0114 rule (_cluster_medoid_position; over-cap path _all_usable_medoid_position; a singleton is its own medoid). The representative tuple (vector, clusterSize, medoidFaceAnnotationId) keeps its shape; only the vector's provenance changes (a real, gate-passing face instead of a synthetic mean). The pairwise distance matrix is already materialized for the linkage in _decompose_person_rows, so the medoid costs one row-sum per cluster — no extra O(n^2). v1–v5 keep centroids byte-identically. Why distance-sum and not distance-to-cluster-centroid: distance-sum is (a) the rule the CV above measured, (b) already the production ADR 0114 rule with a shared, parity-pinned implementation, (c) centroid-free — a centroid-relative variant would be a third definition reintroducing the synthetic point we are removing. Stability under feed churn: the medoid is a deterministic function of the cluster membership set, which is a deterministic function of the (face-id-sorted) input set; the fingerprint hashes exactly that input set, so any membership change flips the digest and recomputes the person — the cache story holds (see V4 for the one real gap).
V2 — Scoring unchanged: ADR 0106 pessimistic per-representative margin stays verbatim. d_pess = d + Z·σ·√(1/n_source + 1/n_target) with n_target = the matched medoid's cluster size; the person key is the min over representatives; the payload distance is the raw distance of the pessimistically chosen representative; shortlist/isolation/ambiguity (ADR 0103/0105) keep raw formulas on the raw-min rep. The measured table keeps medoid-pess at or above plain medoid on wrong-suggest at every m (m=1: 2.92 vs 3.15) at a hair of top-1 — the trade ADR 0106 was designed for. The tempting simplification ('a medoid is a real face, distance to it is directly interpretable, drop the margin') is rejected on the measurement: the margin still buys wrong-suggest reduction exactly where the incident class lives. Cluster size as n_target remains the honest uncertainty proxy for 'this mode has n supporting faces'. Envelope, wire format, and contract version are untouched.
V3 — Fold into semisupervised-v6 by amending ADR 0119; no v7, contract stays 5. The version-per-change discipline (ADR 0116) exists to protect persisted candidate comparability and rollback for methods production has run. v6 is merged but no domain has ever selected it — no v6@5 envelope, candidate set, or stamp generation exists anywhere — so redefining v6's representative kind breaks nobody's comparability, and the rollback story ('stay on v5') is unchanged. Shipping a v7@6 for a definition no production domain ever ran burns a contract generation and doubles registry/matrix/golden surface for zero safety. Precedent: ADR 0108 itself redefined ADR 0104's v2 representation in place before v2 was the default. Guard: the folding PR asserts no discovery_domains.clustering_method = 'semisupervised-v6' row exists (operator confirms; if one ever did, a v7/contract-6 split becomes mandatory). v6 golden fixtures regenerate (the method is dark-shipped). ADR 0119's V2/V3 text gains: 'v6 representatives are per-cluster medoids of the gated selection (ADR 0122)'. Net effect: the operator's v6 A/B decision (0119 V7) evaluates gate + medoid as one switch, not two.
V4 — V168 fingerprint gains a representative-scheme tag (the corrected self-invalidation argument). ADR 0119 V3's argument ('gate the selection before hashing → cache self-invalidates') works because the gate changes the hashed input set. The medoid switch does not change the input set — identical face ids, identical split, identical model pair → identical digest → the cache would HIT and hand back stale centroid representatives. The reasoning must be amended: _representative_fingerprint material gains a scheme line (centroid | medoid) via a new optional parameter defaulting to centroid, so v1–v5 digests are unchanged (their caches never invalidate — byte-identical discipline preserved) and v6 digests carry medoid. Any pre-existing v6 digest (none in production; dev machines irrelevant) mismatches → recompute → correct by construction. This makes the tag a general mechanism: any future representative-rule change bumps the scheme string instead of the cache schema.
V5 — Gate composition (confirms ADR 0119 V1/V3): the anchor-quality gate filters the row selection before decomposition, therefore before medoid selection — a v6 medoid is always a gate-passing real face (or the floor face). The anchor floor extends unchanged: an all-weak person retains its single highest-confidence face, which forms a singleton cluster and is its own medoid, with n_target = 1 → the largest pessimistic margin — conservative in exactly the right direction. Structural consequence worth stating: under medoid representatives, qualification implies there exists a gate-passing face within band — the e3403b1b mechanism (synthetic point qualifies, no real face does) becomes unreachable by construction, independent of any threshold.
V6 — CV protocol (required before production enablement): the gated v6 CV lane (ADR 0119 V6) must sweep representative scheme in {centroid, medoid} × anchor_min_confidence in {off, 0.60, 0.65, 0.70} × clustering_max_distance in {0.35, 0.40, 0.45}, shadow-aware (--include-stashed), k=5 seed=42, importing the production representative build (parity by construction; extend the worker parity test to pin harness medoid_index ≡ production _cluster_medoid_position/_all_usable_medoid_position). Acceptance: G1 gates, zero per-person regressions vs the incumbent ungated [email protected] lane, and the incident fixtures — (a) e3403b1b: with min own-face distance 0.392 > 0.35, no medoid representative may sit within the band (assert min rep distance ≥ min real-face distance, which is the invariant that kills the class); (b) William's 12 wrong tentatives must not qualify at the chosen defaults. The m=1 bucket (single-image add-to-existing) is tracked as a named headline metric in the report, not buried in the aggregate.
V7 — Rollout: inherits ADR 0119 V7 verbatim — dark ship (already merged), operator A/B per domain, mandatory candidate-key churn report which now also covers representative churn (per-person medoid-id changes at switchover). DEFAULT_CLUSTERING_METHOD untouched.
Consequences
Quality: the wrong-person single-add class loses its enabling mechanism (synthetic qualification) and the measured m=1 lane improves +24 pp top-1 / −38% wrong-suggest; larger groups gain ~1–2 pp top-1. Caveat, stated plainly: validated on the owner library only (one household, one embedding space, 745 stashed distractors) — same discipline as ADR 0108's caveat, now with 4.3× the faces.
Costs: (1) one-time representative recompute per (owner, domain, model pair) from the V4 scheme tag — beyond and composing with ADR 0119 V3's gate recompute (a domain switching to v6 recomputes once for both; the decomposition is per-person O(n^2)/O(n^3), household-scale minutes); (2) no wire, envelope, table, or settings changes — the rep tuple shape, person_instance_representatives columns, and ADR 0109 read path are untouched (vector values change; the table is full-replaced per run); (3) golden v6 fixtures regenerate (dark-shipped method).
Thumbnails (ADR 0114 interplay): the cluster decomposition is unchanged by the scheme switch, so thumbnail medoids do not move due to this ADR (they may move under the 0119 gate — that ADR's scope). ADR 0114 and ADR 0108/0119 unify cleanly: the same decomposition pass, the same shared medoid function, the same persisted medoidFaceAnnotationId — after V1, the thumbnail face (largest cluster's medoid) is itself an identity representative, and under v6 every representative is a gate-passing face, so thumbnails inherit the quality gate automatically. Manual pins keep winning for display only and never touch evidence. Display medoid ≠ pessimistically chosen representative may differ — different roles, no conflict.
Risks: a medoid is one sample, not a smoothed mean — for tiny clusters the representative is noisier in principle; the CV says the opposite holds in practice at the sizes that matter (m=1 arrivals, singleton persons are identical under both schemes), and the pessimistic margin absorbs the rest. Medoid choice is embedding-space-specific: a detector/embedding model upgrade shifts which face is most central (revisit trigger). All-weak persons degrade to a single-face representative with the maximum margin — bounded, logged (ADR 0119 floor).
Revisit triggers: per-person regressions or coverage collapse in the V6 sweep; recurrence of wrong-person single-add suggestions under v6+medoid; embedding-model change (re-run the sweep; the V4 scheme tag generalizes if the rule itself changes); ADR 0108's over-cap degradation note re-checked if any person approaches REPRESENTATIVE_SPLIT_MAX_FACES (over-cap v6 persons get one all-faces medoid instead of one all-faces centroid — still one representative, still documented degradation).
Tests. (1) Golden: v1–v5 fixtures unchanged and green (centroid + unchanged digests); v6 fixtures regenerated incl. determinism under permuted feed order and id-sort before linkage. (2) Medoid units: medoid = argmin distance-sum within cluster, tie → smallest face id; singleton = itself; over-cap path uses the all-usable medoid; rep vector is the unit embedding of the medoid face; parity test harness medoid_index ≡ production functions. (3) Fingerprint: v6 digest changes when the scheme tag is added; v1–v5 digests byte-identical; a persisted centroid entry never satisfies a v6 medoid digest (the stale-cache regression this ADR exists to prevent); gate-before-fingerprint behavior from 0119 preserved; floor person's single face is its own medoid. (4) Scoring: d_pess uses the matched medoid's cluster size as n_target; payload distance = raw distance to the chosen medoid; shortlist raw-min over medoid reps. (5) Incident fixtures (V6): e3403b1b invariant min rep distance ≥ min real-face distance, no qualification at 0.35; William's 12 tentatives non-qualifying; m=1 bucket reported. (6) Contract: v6@5 still accepted, no new stamp, envelope shape unchanged; no domain on v6 guard (V3).
Rejected alternatives. (1) v7 as a new method (contract 6) — burns a contract generation and doubles registry/matrix/golden surface for a definition no production domain ever ran; ADR 0108's own precedent (redefining 0104's v2 in place) applies; folding is safe only because v6 was never enabled — guarded in V3. (2) Distance-to-cluster-centroid medoid variant — third medoid definition, reintroduces the synthetic point, breaks parity with ADR 0114's shipped rule; unmeasured. (3) Dropping the pessimistic margin for medoid reps ('distance to a real face is directly interpretable') — measured worse on wrong-suggest at m=1 (3.15 vs 2.92) and the margin is ADR 0106's whole safety story. (4) Centroid-representation toggle inside v6 — another knob doubles the CV matrix and the fingerprint scheme space; v5 already is the centroid rollback. (5) Cheap cache-hit migration re-deriving medoid vectors from persisted medoid ids (no linkage rerun) — correct (same input set ⇒ same clusters ⇒ same medoid ids) but adds a third cache path for a one-time, household-scale-minutes cost; rejected as complexity without payoff. (6) Hashing embedding values or thresholds into the fingerprint — rejected in ADR 0113 (annotations immutable; hash the effect, not the cause); the scheme tag is the minimal correct extension. (7) Percentile gates / other selection rules — inherited rejections from ADR 0119.
Non-goals. No changes to v1–v5 code paths, digests, or representatives. No multi-medoid per cluster (k > 1), no trimmed/weighted means, no new representative kinds. No new operator setting (the representative scheme is part of the method definition; the off-switch is method selection). No scoring-formula changes beyond rep provenance (ADR 0106 margin, ADR 0103/0105 shortlist, envelope, contract). No thumbnail-rule change (ADR 0114 stands; it merely gains the property that its medoid is an identity representative). No default-method change. No re-detection, re-embedding, or quality-score changes.
Links: 0108 — relativized: its centroid-vs-medoid claim was within noise (98.28/1.72 vs 98.26/1.74); remains accepted and unchanged for v1–v5; this ADR is its v6-scoped successor for the representative kind. 0119 — amended: v6's still-unshipped definition gains medoid representatives; V3's self-invalidation argument corrected and extended by V4 (scheme tag); V6's sweep gains the scheme axis; V7's A/B covers both changes. 0114 — the thumbnail medoid rule this ADR reuses verbatim; after V1 the thumbnail face is an actual representative. 0106 — pessimistic margin preserved unchanged, now against real-face representatives. 0113 — V168 fingerprint cache; gains the representative-scheme tag (V4). 0116 — coexistence/contract discipline; V3 argues why folding (not v7) satisfies it for a never-enabled method. 0104, 0101, spec-050 — lineage and qualification semantics.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference