This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
ADR 0116: Identity-aware constrained HAC: U ∪ R clustered together, identity from membership, same-run card merge with runner-up margin demotion (semisupervised-v4)
Status: rejected Date: 2026-10-21
Context
The shipped incumbent semisupervised-v3 (ADR 0113) builds the per-model-pair threshold graph over U ∪ R — the unlabeled pool plus one labeled node per ADR 0108 representative — forms groups on the U×U subgraph ALONE, and derives every candidate's identity from representative ADJACENCY: qualifying(u) = persons with ≥1 representative adjacent to u, wholeGroup = the intersection over members (ADR 0113 V4). Identity evidence therefore influences only the GATES, never group FORMATION: a pool component is clustered without knowing which person its faces sit near, so a face adjacent to two persons' representatives drags its whole group into the empty-wholeGroup review bucket even when the group as a whole sits next to one person, and same-person faces separated in the U×U graph can never pool into one card even where identity was knowable. The completed four-way cross-validation (assignment_cv harness, owner 38a79de6 cache: 2796 faces / 36 persons, arcface-r50/w600k-v1 512d, k=5, seed=42, group sizes 1/2/3/5, band 0.50, threshold 0.45; report v4_comparison_report.md + ADDENDUM, merges to main via PR #1259/#1260) measured the alternative — cluster U and R TOGETHER under the identity-constrained weighted average-linkage HAC and take identity from MEMBERSHIP: aggregated top-1/wrong-suggest 87.09%/8.31% (v1 anchor) · 97.73%/2.14% (v2 medoid-pess) · 98.30%/1.70% with 55 ambiguous (v3) · 98.29%/1.71% with 426 ambiguous (v4) — 6/6 G1 gates PASS, 0 per-person regressions vs v3, Christian→Nathalie and Christian→Jesper wrong-top-1 = 0. The 426 are NOT v3's ambiguity again: split forensics (the ADDENDUM, which REVERSED the first reading) classified the 173 'split' groups as 157 same-person (benign multi-card emission of the CORRECT person), 14 true+others, 1 pure cross-person, 1 no-card — only 50 faces sit in B cards under a wrong name across the whole run, and m=1 coverage EXPANDS (253 v3-ambiguous memberships become card members). The measured runtime 11.38 s vs v3's 4.71 s (2.4×) is a TRANSCRIPT property — the experiment transcribes an O(n²) full-matrix Python loop for clarity (HAC core 7.11 s of 11.38 s, 3112 merges, 887–897 nodes per fold) — while production partitions through hac_linkage's lazy-heap path (ISSUE-100), so the factor is not a production projection. Finally, the queued margin analysis measured the safety net: 50/667 B cards (7.50%) carry runner-up margin < 0.05, of which 16 had a wrong top-1 and 10 had the true person as runner-up — the ADR 0101 ANCHORED_ASSIGN_MARGIN rule (0.05, clustering.py) is cheap and captures exactly the cards plain membership-identity would silently misassign.
LIVE-RUN EVIDENCE FOR THE REVISION (2026-10-19): the first live v4 run (job fc05c137, pool 21194 / anchors 6500) ran ~31 minutes of HAC work — ~4.4× v3's ~7-minute class — and failed only at result submission (ISSUE-169: margin-demoted cards omitted the envelope-required compatiblePersonIds array; fixed worker-side, PR #1315, merged before this revision — part of this baseline). The blowup matches the component-coarsening analysis: medoid nodes bridge previously separate U×U components into GIANT U∪R components, and the HAC's O(n²·d) argmin cost is quadratic in INSTANCE size, so v4 pays (Σ mⱼ)² over coarse components where v3 pays the fine sum Σ nᵢ². Formally, merging two components a and b into one multiplies their combined HAC cost by up to 2×: (a+b)²/(a²+b²) ∈ [1, 2], maximized at a = b — coarsening never pays for itself.
Decision
V1 — New registry method, coexistence, no default change: clustering.py registers SEMISUPERVISED_V4_METHOD = "semisupervised-v4" in CLUSTERING_METHODS and SINGLE_PASS_METHODS alongside v1/v2/v3; the owner selects it per domain via discovery_domains.clustering_method (PeopleClusteringResetController allowlist, Core processing-mode routing and the WorkerResultService isSemisupervised matrix gain v4). contract_version_for_method stamps contract version 3 for v4 (v1/v2 keep 1, v3 keeps 2 — membership identity with card merging is again an identity-structure change, so v4 candidates are comparable only within their own contract generation); WorkerJobEnqueueService snapshots version 3 for v4 jobs, the worker refuses v4@1/v4@2 via the existing unsupported_clustering_contract path, and Core's WorkerResultSemisupervisedEnvelope accepts v4@3 while rejecting v4@2, v4@1, and any v4-stamped envelope against a non-v4 job payload. v1, v2 and v3 stay byte-identical and remain selectable; DEFAULT_CLUSTERING_METHOD is untouched; the rollback is staying on (or returning to) v3 — v3 code is never modified by this ADR.
V2 — Weighted HAC substrate (hac_linkage): the production partition gains per-node multiplicity weights — partition_hac_components, _heap_component_partition and reference_hac_partition accept an optional weights list (default all-ones ⇒ byte-identical behavior; v2 AND v3 share these functions, so the default path is pinned by the existing parity/golden fixtures). Cluster sum vectors compose with weights: S = Σ weight·unit, |a| = Σ weights, average-linkage distance d(a,b) = 1 − (S_a·S_b)/(|a|·|b|) — the exact weighted mean over cross pairs, so a representative of an n-face ADR 0108 cluster counts as n unit faces; S composes under merges, hence the lazy-heap O(n)-memory profile and the (d, i, j) heap tie order of ISSUE-100 are untouched. Weight NEVER enters merge tie-breaking (verified finding of the CV design note: at equal distance the merge order is index-determined; pinned by the expansion-equivalence fixture, not by a tie rule). The identity-conflict rule needs no change: blocking is label-bitmask-only in the existing path and labels never mix under merges, so a pair blocked at init stays blocked — weight changes nothing there; representatives of different persons remain non-mergeable exactly as anchors of different persons are today.
V3 — Identity from membership (the gate change): per model pair the node set is still U ∪ R (_build_rep_graph reused verbatim: U×U + U×R blocks, R labeled with its person id, R never a group member, never consumed, never in a payload), but the connected-component decomposition and HAC run over the FULL U∪R adjacency with labels = person id (R) / None (U) and weights = cluster_size (R — the ADR 0108 cluster size already carried in model_representatives) / 1 (U), replacing v3's U-only strip plus adjacency gates. v3's qualifying(u)/wholeGroup machinery is superseded FOR v4: _semisupervised_v4_candidates does NOT call _emit_group_candidates (which stays untouched so v1/v2/v3 keep byte-identical gates) and reads identity per cluster from membership: representatives of exactly ONE person present with ≥1 U face → B card; no representatives → C (≥2 U faces) / A (1 U face); representatives of ≥2 persons → impossible by V2 blocking (defensive review-only card); representatives only (0 U faces) → no card, logged for transparency, nothing consumed. ADR 0113 V4 remains the shipped gate semantics OF v3; this ADR supersedes it for v4 only. The SCALABLE_MAX_HAC_NODES fail-closed cap carries over unchanged (v4 HAC input is |U|+|R| nodes).
V4 — B-card scoring (the shipped medoid-pess semantics, card-exact): winner medoid = first-minimal d_pess among the card's own medoids in node order; d = cosine between the card's U-face unit centroid and the medoid; n_source = the card's U-FACE COUNT, not the v1/v2/v3 group size m — the card pools all its U faces (clamp ≥ 1); d_pess = d + Z·σ·√(1/n_source + 1/size_rep) with the shipped σ = 0.14, Z = 1.645 (ADR 0106); payload distance = the RAW distance of the pessimistically chosen medoid; the suggest band (suggest_max_distance) classifies on RAW; confidence = clamp(1 − distance); the ADR 0103 shortlist keeps the raw-min over the card's medoids (isolation/ambiguity stay uncorrected raw features).
V5 — Same-run same-target card merge (emission contract, ships WITH v4): identity from membership emits one card per CLUSTER, so a person whose faces form several visual modes can yield several B cards per run/model-pair — the multi-medoid v2 UX consequence, measured as 157 of the 173 split groups (benign multi-card emission of the correct person). v4 therefore merges, within one run and model pair, every B card carrying the same existingInstanceId into ONE card: union of the face lists (deduplicated, ascending faceAnnotationId), union of the contributing clusters' medoids, best (minimum) winner d_pess and its raw distance, occurrence lists concatenated in ascending cluster-key order. Merging happens after HAC and before consumption and margin bookkeeping, so each face is still consumed exactly once per run and churn-suppression sees one card per identity per run (cards key per existingInstanceId). Determinism: permuted feed order → byte-identical candidates (fixture-pinned); the merge never crosses model pairs.
V6 — Runner-up margin demotion ships WITH v4: every B card computes margin = d_pess(runner-up) − d_pess(winner), the runner-up being the min over every OTHER person's best medoid in the pair's FULL train representative universe (first-minimal per person, persons iterated ascending, strict <) — the full universe, not cluster-adjacent identities, so the rule answers exactly what plain medoid-pess ranking would have flagged ambiguous. margin < 0.05 (RUNNER_UP_MARGIN_THRESHOLD — the ANCHORED_ASSIGN_MARGIN precedent, ADR 0101, clustering.py) demotes the card to review-only with the existing ambiguous-card shape: reviewOnly + ambiguous = true, conflictingPersonIds = [winner, runner-up]. Measured cost 7.50% of B cards (50/667); measured benefit 16 wrong top-1s captured plus 10 true-is-runner-up cases. v1/v2/v3 margin semantics are untouched.
V7 — Component-scoped weighted HAC with highest-valid-cut (revision, 2026-10-19): the live-run evidence above (fc05c137, ~31 min at pool 21194) overturns the V3 wording "connected-component decomposition and HAC run over the FULL U∪R adjacency" and its fixed cut at clustering_max_distance — those are SUPERSEDED for v4 by this block. v4 now keeps v3's FINE component decomposition over the U×U subgraph and runs the identity-aware weighted HAC PER U×U COMPONENT, over that component's nodes plus ONLY the representatives ADJACENT to it (rᵢ ≪ nᵢ) — the per-component result equals the global U∪R HAC restricted to the component whenever those representatives do not themselves bridge two fine components (fixture-pinned), so membership identity is preserved wherever it was well-defined and instance cost returns to Θ(d·Σ(nᵢ+rᵢ)²) ≈ Θ(d·Σ nᵢ²) (rᵢ ≪ nᵢ), v3-class. The fixed cut is replaced by the HIGHEST-VALID-CUT: build the component's full dendrogram, then cut at the highest merge height h* ≤ clustering_max_distance at which every resulting subset carries at most one identity label (formally: h* = max{ h ≤ clustering_max_distance : cut(D, h) has no subset with ≥ 2 identity bits }; a linear scan over the merge list with bit-OR identity tracking, O(k) after the HAC); if no valid height ≤ clustering_max_distance exists, cut at the DEEPEST valid height anywhere in the dendrogram (over-fragmentation is acceptable — fragments become separate cards) and the mixed remainder subsets surface via the existing V6 margin demotion. The mixed single-height case degenerates to it: a mixed component with a dendrogram gap below 0.45 separates cleanly AT the gap; a mixed component without any gap → deepest valid cut; a one-identity component keeps the cut at clustering_max_distance (no behavior change). Boundary, accepted: a component with many adjacent representatives (dense multi-person region) inflates its own instance to (nᵢ+rᵢ)² — absorbable and recorded, not gated. Honest runtime statement: v4-revised runtime ≈ v3 runtime — NO speedup vs v3; the gain is QUALITY (mixed components split into correct per-identity cards instead of collapsing into ambiguous mass, m=1 coverage expansion unchanged) and cut depth rises ONLY where identities demand it. Contract stays version 3 and V5 same-run card merge + V6 margin demotion are UNCHANGED — the revision is internal grouping, not envelope shapes; v1/v2/v3 remain byte-identical. Acceptance precondition: the G-A CV lane re-run (validation plan G-A′) before this ADR leaves proposed.
V8 — Rollout: dark ship (registry, allowlist, matrix and stamp additions only), then operator A/B per domain — a v3 → v4 → v3 cycle with a candidate-key churn report — then any default consideration. In-flight safety is inherited: jobs snapshot settings immutably (ADR 0104 D6), a mid-stream switch completes and persists with frozen method/version stamps, and switch-back is a mandatory test case.
REJECTED after exhaustive measurement (2026-10-21). Six variants (fixed-cut, V7-blocked, V7-cut, V8-stream, V9a-kmeans, V9b-1NN) were implemented in the assignment_cv harness and measured over identical folds (k=5, seed 42, group-sizes 1,2,3,5) on TWO ground-truth bases: the original cache (2796 faces / 36 persons, confirmation-biased toward easy suggestions) and an honest re-dump (5764 faces / 42 persons, hard suggestions assigned), each at clustering_max_distance 0.45 and 0.50. NO variant dominates the shipped v3 (representative density graph, ADR 0113). Honest basis, m=1, n=5762: v3 94.6% top-1 / 5.4% wrong-suggest; V7-cut 99.6% / 0.4% at only 33% coverage (precision specialist, coverage problem); V8-stream 90.1% (worse); V9a kmeans 91.7% (worse); V9b 1-NN 94.4% (tie). Key measured facts: highest_valid_cut is a global prefix cut — one early mixed merge freezes the component's replay and discards legal pure merges (no_medoid = 4700/5451 groups at 0.5); stream closures fire at median h'=0.437 (deep, displacing faces); k-means centroid drift costs 2.9 pp vs v3 on the honest basis; 0.45 beats 0.50 for every lane measured. Shipped state remains v3 @ 0.45 (split 0.50).
Consequences
Quality (CV, identical cache — the gate is the LANE-TO-LANE DELTA, absolute numbers optimistically biased by near-duplicate leakage): v4 98.29% top-1 / 1.71% wrong-suggest vs v3's 98.30/1.70 — parity within noise; 6/6 G1 gates PASS with 0 per-person regressions; Christian→Nathalie/Jesper wrong-top-1 = 0. Coverage moves both ways: 253 v3-ambiguous memberships become card members, while 371 v3-ranked groups fall out of ranking (155 split, 216 medoid-less) — the ADDENDUM forensics re-classifies that split mass as 157 same-person benign + 14 true+others + 1 pure cross-person + 1 no-card, only 50 faces under a wrong name, which the V5 merge and the V6 margin rule directly address. The margin rule adds review-only traffic on 7.50% of B cards, capturing 16 wrong top-1s. Costs: (1) churn — group formation is identity-aware, so candidate keys change vs v3, and cluster boundaries now depend on curations: an anchor added or removed changes the representative set and can deterministically move U faces between cards; churn is EXPECTED at curation time, the A/B churn report is mandatory before any default change, and already-persisted v3 candidates are never retro-affected (results persist per job stamp). (2) Runtime — the measured 2.4× transcript factor must NOT be projected onto production: the transcript is an O(n²)-matrix Python loop (HAC core 7.11 s of 11.38 s) while production extends the ISSUE-100 lazy-heap path with weights (two O(n·d) matvecs per chain step, O(n) memory); validation gate G-C bounds the real number (steady-state full run ≤ v3's measured 6 min 16 s + 20% on the deploy host). REVISED BY V7 (2026-10-19): the live fc05c137 run disproved the fixed-cut cost model — coarse U∪R instances paid (Σ mⱼ)² where v3 pays Σ nᵢ² — so the production expectation is v3-CLASS runtime (Θ(d·Σ(nᵢ+rᵢ)²) ≈ Θ(d·Σ nᵢ²), rᵢ ≪ nᵢ), NOT the 2.4× transcript factor and NOT a speedup over v3; the G-C bar is unchanged (≤ 6 min 16 s × 1.20 ≈ 7 min 32 s end-to-end on the deploy host); the largest production component (~13.6k faces) costs the same instance work as v3, so the ADR 0113 <60 s steady-state numeric bar stays DEFERRED, unchanged from 0116 as first drafted. (3) Shared-substrate regression risk — v2/v3 call the same hac_linkage functions, so weights default to 1 and byte-identical default behavior is a fixture-pinned requirement, not an assumption. (4) One more (method, contractVersion) matrix row on both worker and Core. (5) Review traffic moves twice — the V5 merge reduces same-person card counts while V6 adds review-only demotions; the net direction is measured only by the deploy A/B (G-D). Honest limits: no bulk-import win (U ≈ N unchanged — the ANN-indexing follow-up recorded in ADR 0113 stands, unscheduled); the feed remains full-library (feed-level incrementality stays future work); run-to-run grouping now depends on curations — deterministic for identical input, but curation changes legitimately move boundaries; and the CV was measured on ONE library (2796 faces / 36 persons) — the owner-domain deploy A/B is the real gate. REVISION EXPECTATION (V7, 2026-10-19): the CV lane must be RE-RUN for the component-scoped highest-valid-cut lane (validation plan G-A′) — same 6 gates, plus the fixed-cut ambiguity mass collapsing (426 measured → target ≤ 100), same-person splits remaining the benign majority, and harness runtime ≤ 2× the v3 lane; the quality and 426-ambiguity figures above describe the SUPERSEDED fixed-cut lane and stand as the measured baseline the revision improves on, not as the acceptance numbers.
- Measurement artifacts: harness branch feat/cv-highest-valid-cut-v2 (453 tests green), v45_comparison_report.md (all matrices incl. the honest-basis re-measurement).
- If the product ever wants higher precision at lower coverage, V7-cut is the measured candidate; its production gap (component-scoped weighted HAC in _semisupervised_v4_candidates) was never implemented — harness only.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference