6 ADR 0113 representative graph validation plan
Forgejo Actions edited this page 2026-10-16 20:30:43 +00:00
This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

ADR 0113 validation plan — representative density graph (semisupervised-v3)

Status: gate document for ADR 0113 (docs/adr/0113-representative-density-graph.json, status accepted, owner decision 2026-10-16). Gates disposition: G1 PASS (5/6 rows, G1.6 accepted as a documented small-n exception by the owner); G2 PASS (worker fixtures green); G3 PASS (measured, filled from the prototype lane); G4 PARTIAL — design confirmed live (6 min 16 s end-to-end at 40× the gate pool, ~2.5× vs v2 on the same library), the < 60 s numeric bar DEFERRED to post-V6 steady state (V6 representative caching is the recorded follow-up); G5 verified (dark ship, operator A/B on the owner domain, rollback = stay on v2).

Baselines: ADR 0108 CV numbers (top-1 97.0% / wrong-suggest 2.7% at split 0.50, owner library, k-fold 2796 faces / 36 persons) and current semisupervised-v2 production behavior (workers/people/src/memories_people_worker/clustering.py).

Convention note: the repo keeps feature validation under specs/NNN-*/validation/; this ADR has no spec dir (yet), so the plan lives beside the ADR JSON in docs/adr/ and moves into a spec validation dir if a spec-050 amendment spec is spawned at implementation time.


G1 — Quality non-inferiority on the owner library (assignment_cv harness)

Harness: workers/people/experiments/assignment_cv/run_cv.py (experiment code, never imported by the worker), extended with a v3 gate lane — new flag --graph-mode {anchor,representative} (to-create; default anchor reproduces the ADR 0108 baseline bit-identically). The representative lane derives, for every evaluated fold group, per-member qualifying identity sets from U×R representative adjacency at the job's maxDistance (fold-train representatives built by the production _person_representatives transcription, split 0.50), intersects them into the wholeGroup identity, and ranks whole-group identities by the existing medoid-pess rule (ADR 0106 margin, n_target = winning cluster size; parity pinned by workers/people/tests/test_assignment_cv_medoid.py). Groups whose wholeGroup is empty are counted out of ranking exactly like the baseline's out-of-band cases and reported separately (ambiguous-v3 bucket), mirroring the review-only demotion they would get in production.

Protocol: identical folds, group sizes and seed as the ADR 0108 baseline run — both lanes consume the SAME JSONL embedding dump (run_cv.py --load/--dump), so lane-to-lane deltas isolate the graph change. No new dependencies.

Gates (evaluated-group denominator, same as baseline):

Metric Gate
top-1 (representative lane) ≥ 96.5% (97.0% − ε, ε = 0.5 pp)
wrong-suggest (representative lane) ≤ 3.2% (2.7% + ε, ε = 0.5 pp)
Christian → Nathalie wrong-top-1 < 3 (ADR 0108 clause; must not return)
Christian → Jesper wrong-top-1 < 3 (ADR 0108 clause; must not return)
Christian hub-score (wrong top-1 across all persons) ≤ baseline + 2 (ADR 0108 CV remainder watch)
per-person regression no person loses > 2 pp top-1 vs baseline without an offsetting wrong-suggest decrease; regression table required in the report

MEASURED (G1 lane, 2026-10-16, g1_gate_report.md on main, cache of 2026-10-09): 5/6 gates PASS. top-1 (agg) 98.30% PASS · wrong-suggest (agg) 1.70% PASS · Christian→Nathalie 0 PASS · Christian→Jesper 0 PASS · hub-score 23 ≤ 35+2 PASS · per-person regression FAIL — one person (Jutta, n=19 groups): −4.33 pp top-1 with wrong-suggest +4.33 pp, caused by 2 previously-correct groups demoting to the ambiguous-v3 bucket (out of ranking → review-only traffic, not wrong suggestions). The multi-representant representation itself (ADR 0108, already shipped) already costs this person ~7 pp vs the pre-0108 anchor lane — the graph gate adds the demotion on top. ambiguous-v3 bucket overall: 55/3003 = 1.83% of groups. Adjudication (owner, 2026-10-16): ACCEPTED as a documented small-n exception — the 2 demoted groups become review-only traffic, never wrong suggestions, at n=19 groups; G1 therefore PASSES. The anchor lane reproduces the published 86.3%/8.7% pre-0108 baseline exactly, so the protocol is aligned. Remaining open gates at this point: G4 (deploy-host run).

Honesty clause (from the assignment_cv README, restated for this gate): the CV measures the assignment stage with production-parity transcription and is optimistically biased by near-duplicate leakage; the gate is the LANE-TO-LANE delta on identical inputs, not the absolute numbers.

G2 — Synthetic behavioral fixtures

New test file: workers/people/tests/test_representative_graph_clustering.py (existing files to stay green byte-identically: test_semisupervised_clustering.py, test_semisupervised_v2_clustering.py, test_multi_representant.py, test_clustering_golden.py, test_constrained_forward_clustering.py, test_clustering_registry.py, test_target_uncertainty_scoring.py).

Required fixtures:

  1. Edge-existence invariants: node set is exactly U ∪ R (assert no anchor face id is a graph node); only U×U and U×R edges exist; two representatives of different persons are never adjacent (cannot-link broadcast); an anchor face within threshold of a pool face produces NO edge (the documented v2→v3 behavior change — pinned as expected).
  2. Identity separation: a pool face adjacent to representatives of two persons yields both qualifying ids, empty wholeGroup, and the ambiguous review-only demotion (carry-over of test_density_bridge_yields_only_ambiguous_weak_suggestion semantics); distinct identities never share a B card.
  3. Gate-from-representative: a pool face within threshold of a raw anchor but beyond from all of that person's representatives loses its qualifying edge (distortion case, expected); the converse (adjacent to a representative but not to any raw anchor) still qualifies — the edge v3 exists to keep.
  4. Group formation: groups are formed on the U×U subgraph only — a representative bridging two U clusters that are NOT U×U-connected yields two groups (component over U only); SCALABLE_MAX_HAC_NODES fail-closed still fires above the cap.
  5. Determinism: permuted feed order → byte-identical candidates; two runs on identical input → byte-identical (mirrors test_byte_identical_reruns).
  6. Method/contract matrix: v3 registered and in SINGLE_PASS_METHODS; stamps semisupervised-v3 + contract version 2; worker refuses v3@1 and v1/v2@2 via the existing unsupported_clustering_contract path; v1/v2 stamps remain version 1 (extends test_v2_contract_version_is_unchanged).
  7. Core side: WorkerResultSemisupervisedEnvelopeTest gains v3@2 accept / v3@1 and v3@3 reject cases, plus rejection of a v3-stamped envelope on a job that requested v2 (payload/envelope method mismatch); the PeopleClusteringResetController allowlist test covers semisupervised-v3; WorkerJobEnqueueService snapshots version 2 for method v3.

G3 — Performance benchmarks, three regimes

Harness: new workers/people/experiments/rep_graph_bench.py (experiment code), reusing the ISSUE-081 scale-test pattern from test_graph_build_stays_bounded_at_household_scale — synthetic 512-dim embeddings, identical inputs fed to the v2 baseline path (full _density_graph over L+U with labels) and the v3 rep-graph builder. Metrics per run: graph wall time, peak RSS, edge count, connected-component size histogram, HAC partition wall time, and an assertion counter for computed pair blocks (structural proof that no L×L / L×U / L×R block is computed).

Regimes (R sized from a real V156 snapshot or synthetic at household scale, ~1.5k reps):

Regime Composition Gates
Steady state U=500, L≈12k pairs ≤ U²/2 + U·R (asserted); graph wall time ≥ 20× faster than baseline; peak RSS ≤ baseline; MEASURED (prototype lane, synthetic, experiments/rep_graph/benchmark.py): 111× faster (3.35 s → 0.03 s), pairs 78.1M → 342k (228×), peak RSS 733.5 → 8.5 MiB, edges 151,144 → 1,128, max component 102 → 10 (R=434)
Mid U=5k, L≈12k wall time ≥ 3× faster than baseline; peak RSS ≤ baseline; MEASURED: 21.7× faster (9.78 s → 0.45 s), pairs 144.5M → 14.7M, peak RSS 775.5 → 350.0 MiB, max component 113 → 20
Bulk U≈N≈22k no regime may regress vs baseline by > 10% (U·R overhead included); component-size distribution and edge counts reported; MEASURED: 5.7× FASTER (19.58 s → 3.44 s, no regression), peak RSS 818.6 → 713.5 MiB, max component 30 → 30; bulk runs uncapped at full 22k scale

Benchmark provenance and honesty notes: synthetic 512-dim generator with the ADR 0108 geometry (intra-mode 0.280 / cross-mode same-person 0.519 / cross-person 0.999; reps via production _person_representatives at split 0.50); identity sanity on the prototype graph: 100.0% of known-person U faces adjacent to ≥1 own-person rep, 0.0% adjacent to any foreign rep, 100.0% same-mode U–U pairs connected (cross-mode U–U pairs fragment below threshold BY DESIGN — the reps carry identity, the graph carries co-membership). tracemalloc peak scales with U in BOTH variants (per-row embedding conversion), so the prototype introduces no new memory class. The G4 on-host full-run number and the G1 owner-library CV lane remain the outstanding quality/latency gates — these cells settle the STRUCTURAL performance question only.

Component-size histograms are informational (no shape gate) — they feed the churn assessment, not a pass/fail. The household-scale memory guard (test_graph_build_stays_bounded_at_household_scale) must additionally cover the rep-graph builder.

G4 — Run-time gate (16 GB deploy host)

A full steady-state owner_face_recognition run on the owner library (~500 new faces, ~12k anchors, method semisupervised-v3, warm representative cache) must complete < 60 s from feed fetch to result submit, sustained CPU bounded, no OOM — versus 10–14 min today (ISSUE-133). Measurement: worker run timing logs on the deploy host. MEASURED (2026-10-16, deploy host, owner domain switched to semisupervised-v3): first fully successful v3 run 75770b0e — 2875 candidates, HTTP 202, end-to-end 6 min 16 s (20:32:45 → 20:39:01) at pool=20151 / anchors=6350 / tentative=388. The gate's steady-state scenario (~500 new faces) does NOT exist on the live library today: outcome-A faces stay unassigned by design, so ~20k unassigned faces sit in the pool every run — the < 60 s numeric bar is therefore DEFERRED. Measured evidence at 40× the gate pool shows ~2.5× end-to-end speedup vs the same library's v2 runs the same day (11–18 min), consistent with the G3 regime table. Remaining steady-state blockers are the known follow-ups: V6 representative caching (the per-run O(Σ n³) person decomposition is still paid in full) and pool shrink as the owner confirms faces. Verdict: G4 PARTIAL — design confirmed live, numeric gate deferred to post-V6 steady state. Two live-launch seams were found and fixed on the way (ISSUE-164 envelope contract version, ISSUE-165 releasedTentativeFaces canonical-UUID filter); one benign tail heartbeat 403 observed (job completed successfully regardless). Sub-gate (unit-test level): on a fingerprint hit, the worker must not invoke per-person linkage — monkeypatch counter asserts zero cosine_distance_matrix calls for fingerprint-hit persons and full recompute on fingerprint miss (face added/removed, split threshold changed, model pair changed) — in test_representative_graph_clustering.py (or an extension of test_multi_representant.py), plus the G2 warm-cache/no-cache candidate parity.

Post-V6 live measurement (2026-10-16, PRs #1268/#1269): V6 shipped dark and deployed; the unit-level sub-gate is green (zero linkage calls on hits, full recompute on miss, byte-identical hit parity). Live end-to-end pair at pool≈20k: MISS run 50aa2420 6 min 45 s (computes + persists 987/987 fingerprints), HIT run 01f5fcf7 7 min 04 s (same 2919 candidates, HTTP 202) — the delta is WITHIN RUN-TO-RUN NOISE, no measurable end-to-end saving at this library composition. Two honest reasons: (1) run 1's own results perturb assignments (auto-accepts + tentative releases), so run 2 carries per-person fingerprint misses anyway — the cache pays off only across assignment-STABLE runs; (2) the per-person decomposition at the observed person-size distribution (biggest person ~1142 faces, under the 2000 cap) is tens of seconds, not the minutes the projection assumed. The remaining per-run budget is dominated by the HAC partition over the ~20k pool + feed fetch + candidate persistence, none of which V6 touches. Net: V6 removes a real but small cost and pins the mechanism for the incremental-feed future; the dominant levers for the <60s bar stay pool shrinkage (owner confirmations) and feed-level incrementality.

G5 — Rollout checklist

  1. Dark ship: registry + SINGLE_PASS_METHODS + Core processing-mode matrix + PeopleClusteringResetController allowlist + enqueue version stamp only; DEFAULT_CLUSTERING_METHOD unchanged; a production owner stays on v2 until the operator opts in. A v3 result envelope against a non-v3 job payload is rejected (G2.7).
  2. Operator A/B: per-domain switch via settings.clusteringMethod; before any default consideration, run a v2 → v3 → v2 cycle on the owner library and report candidate-key churn (new/changed/removed keys, review-card delta).
  3. In-flight safety: jobs snapshot settings immutably (ADR 0104 D6) — a mid-stream switch completes and persists already-queued jobs with their frozen method/version stamps; switch-back is a mandatory test case.
  4. Suggest-band recalibration is DEFERRED to post-deploy measurement (representative-adjacency distances shift the in-band rate; mirror of the ADR 0108 deployment note) — no suggest_max_distance change ships with v3.
  5. Acceptance: ADR 0113 proposed → accepted only after G1–G4 pass; the TBD (prototype lane) cells are filled at reconciliation; any gate failure returns the ADR to drafting with the measured numbers attached.

Reconciliation

The prototype lane (parallel work) produced the G3 structural numbers (filled above) and the G4 projection. At reconciliation remains: run the G1 owner-library CV lane, measure G4 on the deploy host, record the benchmark artifacts path (report JSON/markdown, never embedding caches), update the ADR status if all gates pass, and file KNOWN_ISSUES entries for any documented degradation accepted on the way (e.g. representative edge-flip cases observed at owner scale).