feat: semisupervised-v2 — constrained average-linkage grouping (ADR 0104) #1035

Merged
chris merged 1 commit from feat/semisupervised-v2-average-linkage into main 2026-10-05 08:50:32 +02:00
Owner

What

Implements ADR 0104 (accepted 2026-10-05): a new opt-in clustering method semisupervised-v2 — constrained standard HAC average linkage replaces v1's greedy complete-linkage clique growth. semisupervised-v1 stays byte-unchanged; the default method stays constrained-dbscan. Zero behavior change unless an owner switches their people domain to semisupervised-v2 (admin UI → discovery domain → clustering method).

Design invariants (from ADR 0104)

  • Same contract: envelope clusteringContractVersion stays 1 for both methods; no DB migration; same candidate shapes, same ADR 0103 qualityScore, same A/B/C gates.
  • D3 wholeGroup preserved: identity evidence is read from the threshold graph (every member needs a within-threshold confirmed anchor of the target identity), never from HAC group membership. Ambiguity → review-only card, unchanged.
  • Identity safety: explicit group-level confirmed-identity conflict rule before every merge — under average linkage, removing cannot-link edges alone would NOT prevent a mean ≤ threshold merge across identities.
  • Not a linkage swap: HAC changes search strategy AND anchor consumption semantics (anchors are partitioned and group-gated instead of reused during growth). Documented in ADR 0104; expected effect is fewer wrong 2-face fragments.

Implementation

Worker (workers/people):

  • hac_linkage.py (new): reference HAC partition (experiment-validated semantics: global best-first merge, mean of ALL cross-pairs, smallest-index tie break, full matrix, hard member cap) + production partition_hac_components — exact connected-component decomposition of the threshold graph (all cross-component pairs exceed the threshold, so the reference could never merge across components). Fail closed: a component above the member cap (512 deployed / 768 in dev checkouts with the experiment harness) aborts the run with ComponentTooLargeError — no silent degradation (tracked as ISSUE-100; D4 scalable-exact algorithm is future work).
  • clustering.py: emission block extracted verbatim into shared _emit_group_candidates; v1 refactored onto it with behavior identical (all golden tests untouched and passing); _semisupervised_v2_candidates reuses the entire v1 outer loop (model-pair partitioning, run-global face ownership, dedup) and swaps only group formation. Candidates stamped clusteringMethod=semisupervised-v2, version 1.
  • main.py: v2 rides the v1 contract-version gate (v1+v2 @ version 1); explicit v2 job with wrong/missing version still fails the job.

Core (apps/api): v2 routes to the same semisupervised processing mode; envelope validation matches the payload's expected method (v1 or v2) against the same version; enqueue snapshots clusteringContractVersion=1 + maxDistance for both; method allowlist extended.

Web (apps/web): semisupervised-v2 option in the admin clustering-method select; the distance control shows for either semisupervised method.

Verification

  • Workers suite: 244 passed (229 existing incl. v1 goldens + 15 new v2 tests: mean-bridge vs greedy fragmentation, identity safety, D3 wholeGroup, component-decomposition parity incl. tie cases on seeded random fixtures, byte determinism, fail-closed cap, gate shapes, handler version rejection)
  • :apps:api:test full suite green; web vitest 65 passed; svelte-check 0 errors
  • Rollout threshold: the stored clustering_max_distance (default 0.35) — recalibration is deliberately out of scope (ADR 0104 D2)
  • Rollback: flip the domain setting back to v1/legacy — future jobs only (in-flight jobs complete; immutable payload snapshots)

Meta

  • ADR 0104 → accepted (user approval in session)
  • Spec 050: new task S050T004 (done, PR #1035)
  • KNOWN_ISSUES: ISSUE-100 (component-cap fail-closed limitation, open)
  • Evidence base: PR #1033 (reviewed synthetic-only comparison; no real-library claims)
## What Implements ADR 0104 (accepted 2026-10-05): a new **opt-in** clustering method `semisupervised-v2` — constrained standard HAC **average linkage** replaces v1's greedy complete-linkage clique growth. `semisupervised-v1` stays byte-unchanged; the default method stays `constrained-dbscan`. **Zero behavior change unless an owner switches their people domain to `semisupervised-v2`** (admin UI → discovery domain → clustering method). ## Design invariants (from ADR 0104) - **Same contract**: envelope `clusteringContractVersion` stays **1** for both methods; no DB migration; same candidate shapes, same ADR 0103 qualityScore, same A/B/C gates. - **D3 wholeGroup preserved**: identity evidence is read from the threshold graph (every member needs a within-threshold confirmed anchor of the target identity), **never** from HAC group membership. Ambiguity → review-only card, unchanged. - **Identity safety**: explicit group-level confirmed-identity conflict rule before every merge — under average linkage, removing cannot-link edges alone would NOT prevent a mean ≤ threshold merge across identities. - **Not a linkage swap**: HAC changes search strategy AND anchor consumption semantics (anchors are partitioned and group-gated instead of reused during growth). Documented in ADR 0104; expected effect is fewer wrong 2-face fragments. ## Implementation **Worker** (`workers/people`): - `hac_linkage.py` (new): reference HAC partition (experiment-validated semantics: global best-first merge, mean of ALL cross-pairs, smallest-index tie break, full matrix, hard member cap) + production `partition_hac_components` — exact connected-component decomposition of the threshold graph (all cross-component pairs exceed the threshold, so the reference could never merge across components). **Fail closed**: a component above the member cap (512 deployed / 768 in dev checkouts with the experiment harness) aborts the run with `ComponentTooLargeError` — no silent degradation (tracked as ISSUE-100; D4 scalable-exact algorithm is future work). - `clustering.py`: emission block extracted **verbatim** into shared `_emit_group_candidates`; v1 refactored onto it with behavior identical (all golden tests untouched and passing); `_semisupervised_v2_candidates` reuses the entire v1 outer loop (model-pair partitioning, run-global face ownership, dedup) and swaps only group formation. Candidates stamped `clusteringMethod=semisupervised-v2`, version 1. - `main.py`: v2 rides the v1 contract-version gate (v1+v2 @ version 1); explicit v2 job with wrong/missing version still fails the job. **Core** (`apps/api`): v2 routes to the same semisupervised processing mode; envelope validation matches the payload's expected method (v1 **or** v2) against the same version; enqueue snapshots `clusteringContractVersion=1` + `maxDistance` for both; method allowlist extended. **Web** (`apps/web`): `semisupervised-v2` option in the admin clustering-method select; the distance control shows for either semisupervised method. ## Verification - Workers suite: **244 passed** (229 existing incl. v1 goldens + 15 new v2 tests: mean-bridge vs greedy fragmentation, identity safety, D3 wholeGroup, component-decomposition parity incl. tie cases on seeded random fixtures, byte determinism, fail-closed cap, gate shapes, handler version rejection) - `:apps:api:test` **full suite green**; web vitest **65 passed**; svelte-check 0 errors - Rollout threshold: the stored `clustering_max_distance` (default 0.35) — recalibration is deliberately out of scope (ADR 0104 D2) - Rollback: flip the domain setting back to v1/legacy — future jobs only (in-flight jobs complete; immutable payload snapshots) ## Meta - ADR 0104 → `accepted` (user approval in session) - Spec 050: new task S050T004 (done, PR #1035) - KNOWN_ISSUES: ISSUE-100 (component-cap fail-closed limitation, open) - Evidence base: PR #1033 (reviewed synthetic-only comparison; no real-library claims)
feat(workers,api,web): semisupervised-v2 — constrained average-linkage grouping (ADR 0104)
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 58s
36d633dbab
Opt-in clustering method `semisupervised-v2`; semisupervised-v1 stays byte-
unchanged. Same envelope contract version 1, same A/B/C outcome gates, same
card shapes and ADR 0103 qualityScore — only the group-forming strategy
changes: greedy complete-linkage clique growth -> constrained standard HAC
average linkage (mean of ALL cross-pair distances, global best-first merge,
stable smallest-index tie break) with the explicit group-level confirmed-
identity conflict rule (removing cannot-link edges alone would not stop a
mean<=threshold merge across identities).

Worker (workers/people):
* hac_linkage.py: reference HAC partition (experiment-validated semantics,
  full matrix, hard member cap, fail closed) + production
  partition_hac_components (exact connected-component decomposition of the
  threshold graph: all cross-component pairs exceed the threshold, so the
  reference could never merge across components; parity fixture-tested).
* clustering.py: emission block extracted verbatim into the shared
  _emit_group_candidates (v1 refactored onto it, behavior identical);
  _semisupervised_v2_candidates reuses the entire v1 outer loop and swaps
  only group formation; D3 wholeGroup rule preserved — identity evidence
  comes from the threshold graph, never from HAC membership.
* main.py: v2 rides the v1 contract-version gate (v1+v2 @ version 1).
* tests: 15 new v2 tests (mean-bridge vs greedy fragmentation, identity
  safety, D3 wholeGroup, component parity incl. ties, byte determinism,
  fail-closed cap, gate shapes); suite 244 passed. v1 golden tests
  untouched and passing.

Core (apps/api):
* WorkerResultService: v2 routes to the same SEMISUPERVISED processing mode;
  envelope validation matches the payload's expected method (v1 or v2)
  against the same contract version — mismatch still fails closed.
* WorkerJobEnqueueService: v2 jobs snapshot clusteringContractVersion=1 and
  maxDistance exactly like v1.
* PeopleClusteringResetController: semisupervised-v2 allowed in the method
  allowlist (per-domain opt-in; default stays constrained-dbscan).

Web (apps/web): method type + admin select option; the distance control now
shows for either semisupervised method.

Meta: ADR 0104 -> accepted (user approval 2026-10-05); spec 050 task
S050T004; KNOWN_ISSUES ISSUE-100 (component-cap fail-closed until the D4
scalable-exact algorithm lands).

Verification: workers 244 passed (PYTHONPATH=src); :apps:api:test full suite
green; web vitest 65 passed; svelte-check 0 errors. No migration, no
envelope change; default method unchanged — zero behavior change unless an
owner switches their domain to semisupervised-v2.
chris merged commit 99c13c22d7 into main 2026-10-05 08:50:32 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1035
No description provided.