2 ADR 0116 identity aware hac validation plan
Forgejo Actions edited this page 2026-10-19 21:22:05 +00:00
This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

ADR 0116 validation plan — identity-aware constrained HAC (semisupervised-v4)

Status: gate document for ADR 0116 (docs/adr/0116-identity-aware-hac.json, status proposed). Gate disposition at drafting: G-A DONE (measured, 6/6 PASS) — the CV evaluation completed BEFORE this ADR was drafted; G-B, G-C, G-D open, to be executed at implementation. Amended 2026-10-19 (ADR 0116 decision V7 — component-scoped weighted HAC with highest-valid-cut, prompted by the first live v4 run, job fc05c137, and its submission failure ISSUE-169, fixed via PR #1315): the measured lane above is the SUPERSEDED global-U∪R fixed-cut design; G-A′ below makes its re-run for the revised lane a mandatory acceptance precondition and G-B gains the V7 fixtures (9–11). G-C and G-D are unchanged. This plan extends the conventions of the ADR 0113 plan (docs/adr/0113-representative-graph-validation-plan.md); read that document first for the harness protocol and honesty clauses.

Baselines: the completed four-way CV (v1 anchor 87.09/8.31 · v2 medoid-pess 97.73/2.14 · v3 representative 98.30/1.70/55-amb · v4 identity-HAC 98.29/1.71/426-amb, aggregated over k=5/seed=42/group sizes 1,2,3/5, band 0.50, threshold 0.45) and current semisupervised-v3 production behavior (workers/people/src/memories_people_worker/clustering.py, v3's measured 6 min 16 s full run on the deploy host, ADR 0113 G4).

Convention note: as with ADR 0113, this ADR has no spec dir (yet), so the plan lives beside the ADR JSON in docs/adr/ and moves into a spec validation dir if an implementation spec is spawned.


G-A — CV non-inferiority on the owner library (assignment_cv harness) — DONE

Protocol: identical folds, group sizes and seed as the ADR 0108/0113 lane runs — all four lanes on the SAME JSONL embedding dump, so lane-to-lane deltas isolate the method change. Gates are the ADR 0113 G1 thresholds applied to the v4 lane, with the per-person regression gate evaluated against v3 (the shipped incumbent), not the anchor baseline.

MEASURED (2026-10-19, committed report workers/people/experiments/assignment_cv/v4_comparison_report.md + ADDENDUM — split forensics — in the .worktrees/v4-cv-eval evaluation checkout; DEPENDENCY: that report and its addendum must be merged to main before this ADR's proposed → accepted reconciliation can reference them from main; merge via PR #1259/#1260):

6/6 gates PASS.

gate threshold measured (v4) verdict
G-A.1 top-1 (aggregated) ≥ 96.5% 98.29% PASS
G-A.2 wrong-suggest (aggregated) ≤ 3.2% 1.71% PASS
G-A.3 Christian → Nathalie wrong-top-1 < 3 0 PASS
G-A.4 Christian → Jesper wrong-top-1 < 3 0 PASS
G-A.5 hub-score vs ADR 0108 medoid baseline ≤ baseline + 2 7 ≤ 35 + 2 (vs anchor baseline 0: reported, structural — see report gate-baseline note) PASS
G-A.6 per-person regression vs v3 no person loses > 2 pp top-1 without offsetting wrong-suggest decrease 0 persons PASS

Ambiguity accounting: v4 426 ambiguous groups (14.19%) vs v3's 55 (1.83%) — NOT comparable raw. Split forensics (the ADDENDUM, which corrected the first reading): of 173 split groups, 157 same-person (benign multi-card emission of the CORRECT person — addressed by the V5 same-run card merge), 14 true+others, 1 pure cross-person, 1 no-card; only 50 faces in B cards under a wrong name; 253 v3-ambiguous memberships become card members. Margin analysis: 50/667 B cards (7.50%) below 0.05, of which 16 wrong top-1 and 10 true-is-runner-up — the measured basis for shipping the V6 margin rule WITH v4.

Honesty clauses carried over: the gate is the LANE-TO-LANE DELTA on identical inputs; absolute numbers are optimistically biased by near-duplicate leakage; the CV was measured on ONE library (2796 faces / 36 persons) — G-D is the real gate. The measured runtime 11.38 s vs v3's 4.71 s (2.4×) is a transcript property of the experiment's O(n²) full-matrix Python loop, NOT a production projection — production performance is gated by G-C.

G-A′ — CV re-run for the V7 lane (component-scoped weighted HAC with highest-valid-cut) — REQUIRED

Why: the measured G-A lane is the SUPERSEDED global U∪R HAC with fixed cut at clustering_max_distance; V7 changes group formation (per-U×U-component weighted HAC with only the representatives adjacent to that component, rᵢ ≪ nᵢ) and the cut rule (highest-valid-cut), so the fixed-cut measurements — including the 426-ambiguity mass — do not describe the shipping design. Protocol: identical folds, group sizes and seed as the measured lane — the revised v4 lane against the same v1/v2/v3 lanes on the SAME JSONL embedding dump, so lane-to-lane deltas isolate the V7 change.

Acceptance (all of):

gate threshold
G-A′.1 the 6 measured-lane gates (G-A.1–G-A.6, per-person regression still vs v3) re-PASS for the revised lane
G-A′.2 ambiguous count drops vs the 426 fixed-cut measurement — target ≤ 100 (the revision's purpose is splitting mixed components into per-identity cards instead of ambiguous mass)
G-A′.3 same-person splits remain the BENIGN majority of remaining split groups (ADDENDUM forensics protocol re-applied to the new run)
G-A′.4 harness runtime ≤ 2× the v3 lane, AND the transcript shows the PER-COMPONENT HAC work (component scoping must be visible in the transcript itself, not only in the total)

Disposition: a G-A′.1 failure returns the ADR to drafting; a G-A′.2 miss is a drafting failure of the quality claim and also returns the ADR to drafting; a G-A′.4 miss means component scoping did not materialize in the implementation — fix the implementation, do not renegotiate the gate.

G-B — Synthetic behavioral fixtures

New test file: workers/people/tests/test_identity_hac_clustering.py (existing files stay green byte-identically, including test_representative_graph_clustering.py and the hac_linkage parity/golden fixtures — the weights=1 default path of the shared substrate is itself a fixture requirement, see G-B.6).

Required fixtures:

  1. Straddle blocking (the ADR 0104 conflict rule under weights): the design-note 2D circle geometry (Anna medoid weight 5 + Anna U faces at 6°/12°, bridge face at 45°, Ben U faces at 84°/90° + Ben medoid weight 3, all chain distances ≤ 0.45) yields exactly TWO identity-pure clusters — the bridge joins Anna first (0.157 < 0.223), then Anna ↔ Ben is blocked; never one mixed cluster. Premise asserted: unconstrained the chain merges.
  2. Weighted expansion-equivalence: expand each weighted representative into weight-many unit copies, run production reference_hac_partition on the expanded set (labels propagate), assert the induced partition matches the compact weighted partition (geometry without exact tie-order divergence). Pins: weight enters through average-linkage geometry only — weight NEVER breaks merge ties (the CV design note's verified finding).
  3. Scoring-weight margin: at equal raw distance, the heavier medoid yields the lower d_pess (runner-up = heavier one) — the V6 rule's direction.
  4. Same-run card merge determinism: two same-person B cards in one run/model-pair merge into one card (union of faces, best winner d_pess/margins, merged occurrences); permuted feed order → byte-identical candidates; the merge never crosses model pairs; each face consumed exactly once.
  5. Margin demotion: a B card with runner-up margin < 0.05 demotes to review-only with reviewOnly/ambiguous + conflictingPersonIds [winner, runner-up]; margin ≥ 0.05 emits normally; runner-up computed over the FULL train medoid universe (first-minimal per person, persons ascending), not just cluster-adjacent identities.
  6. Substrate invariants (weights=1 byte-identity): with default weights, partition_hac_components/_heap_component_partition/ reference_hac_partition produce byte-identical output to pre-0116 runs (v2/v3 golden parity untouched); medoid-only clusters emit no card and consume nothing; SCALABLE_MAX_HAC_NODES fail-closed still fires on the U∪R node count (under V7: on each PER-COMPONENT instance of nᵢ+rᵢ nodes — the V3 global-U∪R wording superseded).
  7. Method/contract matrix: v4 registered and in SINGLE_PASS_METHODS; stamps semisupervised-v4 + contract version 3; worker refuses v4@2 and v4@1 via the existing unsupported_clustering_contract path; v1/v2 stamp 1 and v3 stamp 2 unchanged (extends test_v2_contract_version_is_unchanged).
  8. Core side: WorkerResultSemisupervisedEnvelopeTest gains v4@3 accept / v4@2, v4@1 reject cases, plus rejection of a v4-stamped envelope on a job that requested v3; the PeopleClusteringResetController allowlist test covers semisupervised-v4; WorkerJobEnqueueService snapshots version 3 for method v4.

Fixtures 9–11 pin the V7 revision (component-scoped weighted HAC with highest-valid-cut):

  1. Highest-valid-cut unit tests (the cut rule in isolation, on crafted dendrograms): a MIXED component whose dendrogram has a gap below 0.45 separates cleanly AT the gap (every subset ≤ 1 identity bit, height maximal); a MIXED component with no valid height ≤ 0.45 cuts at the DEEPEST valid height anywhere in the dendrogram, and its mixed remainder subsets surface via the V6 margin demotion (review-only, ambiguous); a SINGLE-identity component keeps the cut at clustering_max_distance (byte-identical to the fixed-cut behavior). Also pinned: the linear bit-OR scan is O(k) after the HAC, and the scan is deterministic under permuted feed order (consistent with the V5 determinism fixture).
  2. Component-scoped HAC equivalence: when no representative bridges two fine components, the V7 per-component result (component nodes + only the representatives adjacent to it, rᵢ ≪ nᵢ) equals the global U∪R HAC RESTRICTED to that component — the equality V7's runtime claim rests on. Counter-case: a representative ADJACENT to two components keeps them in one identity-aware instance (never silently dropped); the component decomposition over U×U matches v3's exactly.
  3. Runtime regression pin: on a fixed synthetic library whose medoids bridge several U×U components, the V7 per-component path does NOT coarsen into a giant instance — total HAC instance cost scales as Σ(nᵢ+rᵢ)², not (Σ mⱼ)² (guards against a regression to the global U∪R shape that produced the fc05c137 ~31-min run); harness-level companion to the G-A′.4 transcript requirement.

G-C — Production performance (weighted partition_hac_components)

The 2.4× transcript factor must NOT transfer to production: production partitions through the ISSUE-100 lazy-heap path (two O(n·d) matvecs per chain step, O(n) memory), where weights compose inside the existing sum-vector arithmetic. Gate: a full steady-state owner_face_recognition run with method semisupervised-v4 on the owner library (deploy host, warm representative cache) completes in ≤ 6 min 16 s × 1.20 ≈ 7 min 32 s end-to-end — i.e. not slower than v3's measured 6 min 16 s baseline by more than 20%. Measurement: worker run timing logs on the deploy host, same-day v3-vs-v4 pair to cancel host noise. Sub-checks (unit level, in the G-B file): the weighted heap path performs the same two-matvec chain-step shape (no pairwise matrix allocated — extend the ISSUE-081 memory guard), and the U∪R component decomposition stays within the ISSUE-100 headroom argument at SCALABLE_MAX_HAC_NODES.

G-D — Deploy A/B on the owner domain

  1. Dark ship: registry + SINGLE_PASS_METHODS + Core matrix + allowlist + enqueue stamp only; DEFAULT_CLUSTERING_METHOD unchanged.
  2. Operator A/B: switch the owner domain v3 → v4 → v3 (mandatory switch-back); per run compare card counts, review-card counts and review-traffic delta vs v3's last runs on the same library; report the candidate-key churn (new/changed/removed keys) — churn is EXPECTED (group formation is identity-aware and depends on curations); the V5 merge keys cards per existingInstanceId, so same-person re-emission within a run collapses before churn accounting.
  3. In-flight safety: a mid-stream switch completes and persists already- queued jobs with frozen method/version stamps (ADR 0104 D6); a v4 envelope against a v3 job is rejected (G-B.8).
  4. Acceptance: ADR 0116 proposed → accepted only after G-B, G-C and G-D pass AND the G-A′ re-run of the revised lane meets G-A′.1–G-A′.4 (the measured G-A lane documented the superseded fixed-cut design; G-A′.1–.4 re-measure it for V7); any gate failure returns the ADR to drafting with the measured numbers attached; file KNOWN_ISSUES entries for any degradation accepted on the way.

Reconciliation

Fill the G-C measured cell from deploy-host logs, the G-D churn/card-count report from the operator A/B runs, record the G-A′ re-run result (gate table plus the forensics ambiguity accounting of G-A′.2/G-A′.3), record the benchmark artifact paths (report JSON/markdown, never embedding caches), confirm the v4 comparison report + ADDENDUM are on main, and update the ADR status if all gates pass.