feat(people): semisupervised-v1 single-pass A/B/C clustering (spec 050, ADR 0101) #996

Merged
chris merged 6 commits from feat/semisupervised-clustering into main 2026-10-03 08:09:52 +02:00
Owner

Summary

Replaces the multi-pass people recognition pipeline (step-1 nearest-anchor emit + clustering + weak band + legacy fallback + Core embedding-match rematch) with a new clustering method semisupervised-v1 implementing the owner's A/B/C contract (ADR 0101, spec 050; root cause of ISSUE-087):

  • ONE semi-supervised complete-linkage clustering pass over confirmed anchors (labeled, cannot-link) + unassigned faces (unlabeled). Every physical face lands in at most one cluster (eligibility-aware growth + model-local maps + run-global ownership; corrected multi-tile graph mode).
  • A isolated singleton stays unassigned, no card. B clique with exactly one identity becomes a tentative occurrence + visible actionable backing card; auto-confirmed only via effective owner policy + identity-score threshold. C unlabeled pair/group (>=2) becomes a new-person suggestion ONLY. Competing identities are ALWAYS review-only (structural, no margin knob).
  • Exactly ONE clustering parameter: persisted clustering_max_distance (V150, default 0.35, CHECK 0-2, 3 decimals) — super-admin GET/PATCH /api/admin/people/clustering-max-distance, admin UI control visible only for the new method. No redundant knobs.
  • Job-bound contract: processing mode derived from the immutable job payload; strict envelope validation before any write (method/contractVersion incl. empty results, canonical UUIDs, duplicates, C shape/score checks, forged stamps rejected); worker fails explicitly on unsupported contract versions — never silent legacy fallback.
  • Card lifecycle: ALLOCATE_TENTATIVE_KEEP_PENDING inside the source-first locked acceptance primitive (no accepted-review fiction; audit-event provenance); backing cards survive same-target replay, partial overlap, automatic confirmation and end-of-job superseding; rejected pairs stay suppressed; owner-clustering reset serialized against result application via owner-scoped tenant lock (V151, fail-closed); ambiguous cards require an explicit owner-chosen target (bulk confirm refuses them).
  • Legacy methods byte-unchanged (worker differential-parity harness; legacy guards are payload-stamp-scoped).
  • Docs: ADR 0101 accepted, spec 050 T001/T002 done, ISSUE-087 mitigated. Versions: web 0.0.40 -> 0.0.41, api 1.21.62 -> 1.21.63.

Verification

  • API: 1481 tests, 0 failures (full suite; incl. 31 Spec050 PG transactional tests with latch-driven reset/apply races, 27 envelope H2 tests incl. golden byte-exact real-worker fixtures, 11 config tests)
  • Worker: 183 passed (142 legacy + 41 new; incl. oracle bridge repro, two-model single emission, anchor-order-invariant ambiguity, multi-tile scalar reference, legacy parity)
  • Web: svelte-check 0 errors, 60 vitest + 32 Playwright green
  • Independent oracle gate: final review APPROVED after the envelope-seam fix (ambiguous-with-score / unscored-no-lists shapes now validated by golden seam tests)

Rollout constraint (important)

Deploy API and people-worker together BEFORE switching the domain method to semisupervised-v1. A skewed (old) worker receiving a semisupervised job fails closed: envelope rejection -> visible processing issue -> bounded retries (3) -> dead job. Never silent legacy processing. After deploy: switch method in Admin (Erkennung), optionally adjust maxDistance, then run the owner clustering reset to recluster with the new method.

## Summary Replaces the multi-pass people recognition pipeline (step-1 nearest-anchor emit + clustering + weak band + legacy fallback + Core embedding-match rematch) with a new clustering method **`semisupervised-v1`** implementing the owner's A/B/C contract (ADR 0101, spec 050; root cause of ISSUE-087): - **ONE semi-supervised complete-linkage clustering pass** over confirmed anchors (labeled, cannot-link) + unassigned faces (unlabeled). Every physical face lands in at most one cluster (eligibility-aware growth + model-local maps + run-global ownership; corrected multi-tile graph mode). - **A** isolated singleton stays unassigned, no card. **B** clique with exactly one identity becomes a tentative occurrence + visible actionable backing card; auto-confirmed only via effective owner policy + identity-score threshold. **C** unlabeled pair/group (>=2) becomes a new-person suggestion ONLY. Competing identities are ALWAYS review-only (structural, no margin knob). - **Exactly ONE clustering parameter**: persisted `clustering_max_distance` (V150, default 0.35, CHECK 0-2, 3 decimals) — super-admin GET/PATCH /api/admin/people/clustering-max-distance, admin UI control visible only for the new method. No redundant knobs. - **Job-bound contract**: processing mode derived from the immutable job payload; strict envelope validation before any write (method/contractVersion incl. empty results, canonical UUIDs, duplicates, C shape/score checks, forged stamps rejected); worker fails explicitly on unsupported contract versions — never silent legacy fallback. - **Card lifecycle**: ALLOCATE_TENTATIVE_KEEP_PENDING inside the source-first locked acceptance primitive (no accepted-review fiction; audit-event provenance); backing cards survive same-target replay, partial overlap, automatic confirmation and end-of-job superseding; rejected pairs stay suppressed; owner-clustering reset serialized against result application via owner-scoped tenant lock (V151, fail-closed); ambiguous cards require an explicit owner-chosen target (bulk confirm refuses them). - **Legacy methods byte-unchanged** (worker differential-parity harness; legacy guards are payload-stamp-scoped). - Docs: ADR 0101 accepted, spec 050 T001/T002 done, ISSUE-087 mitigated. Versions: web 0.0.40 -> 0.0.41, api 1.21.62 -> 1.21.63. ## Verification - API: **1481 tests, 0 failures** (full suite; incl. 31 Spec050 PG transactional tests with latch-driven reset/apply races, 27 envelope H2 tests incl. golden byte-exact real-worker fixtures, 11 config tests) - Worker: **183 passed** (142 legacy + 41 new; incl. oracle bridge repro, two-model single emission, anchor-order-invariant ambiguity, multi-tile scalar reference, legacy parity) - Web: svelte-check 0 errors, 60 vitest + 32 Playwright green - Independent oracle gate: final review **APPROVED** after the envelope-seam fix (ambiguous-with-score / unscored-no-lists shapes now validated by golden seam tests) ## Rollout constraint (important) **Deploy API and people-worker together BEFORE switching the domain method to semisupervised-v1.** A skewed (old) worker receiving a semisupervised job fails closed: envelope rejection -> visible processing issue -> bounded retries (3) -> dead job. Never silent legacy processing. After deploy: switch method in Admin (Erkennung), optionally adjust maxDistance, then run the owner clustering reset to recluster with the new method.
- WorkerResultService: processing mode derived from immutable job payload
  and passed explicitly; strict semisupervised envelope validation before
  any write (integral contractVersion=1, strict Boolean flags, canonical
  UUIDs, duplicate/alias/score checks, discovery-job forge rejection)
- ClassInstanceDiscoveryService: ALLOCATE_TENTATIVE_KEEP_PENDING inside
  the source-first locked acceptance primitive (no accepted review-decision
  fiction, audit_events provenance instead); actionable-subset persistence
  rewrites all membership/evidence/representative state; rejected-pair
  suppression; backing cards protected during tentative allocation,
  automatic confirmation and end-of-job superseding; ambiguity
  explicit-target guard scoped to stamped provenance incl. bulk confirm
- TenantClusteringLockService + V151: owner-scoped transactional barrier
  serializing clustering reset vs result application (fail-closed
  staleness guard)
- V150 + endpoint: persisted clustering_max_distance (default 0.35, CHECK
  constraint), super-admin GET/PATCH /api/admin/people/clustering-max-distance;
  enqueue snapshots method/contractVersion/maxDistance
- Tests: Spec050 27 PG transactional (latch-driven races, replay/overlap,
  next-job export), envelope 21 H2, config 11 PG; full API suite 1468 green
- clustering.py: _semisupervised_candidates — ONE complete-linkage pass per
  model pair over confirmed anchors (labeled, cannot-link) + unassigned
  faces (unlabeled); eligibility-aware greedy clique growth enforces
  disjoint unlabeled membership DURING growth; model-local face maps with
  in-model dedup, anchors preserved per model; run-global physical-ID
  ownership re-checked immediately before each partition's graph build;
  A singleton (nothing), B attach (non-weak, centroid score, reviewOnly
  when unscoreable), C new-person (>=2 members); structural ambiguity
  (qualifying/wholeGroup from the SAME graph, no margin) always
  review-only with conflictingPersonIds/compatiblePersonIds
- _density_graph: optional corrected_tiles mode (cols > start + rows)
  fixing the verified multi-tile self/duplicate-edge bug; legacy default
  unchanged bit-for-bit
- main.py: SINGLE_PASS_METHODS gate skips step-1 computation, weak band
  and legacy fallback; envelope stamped clusteringMethod +
  clusteringContractVersion (also when empty); unsupported/missing
  contract version fails the job explicitly (issue, no candidates)
- Tests: 40 new (182 total green) incl. oracle bridge repro, two-model
  single emission, anchor-order-invariant ambiguity, multi-tile scalar
  reference, legacy differential parity harness
- operatorStatus.ts: ClusteringMethod union + typed GET/PATCH client for
  /api/admin/people/clustering-max-distance (string contract, numeric
  PATCH, explicit null reset)
- AdminDiscoveryTab: fourth method option + one distance section visible
  only for semisupervised-v1 (save/reset/loading/error/success, 404
  degradation, blank input never PATCHes); copy explains stricter vs
  looser similarity without promising exact cluster counts
- Tests: 5 vitest serialization + 14 Playwright (desktop+mobile);
  admin-operator regression green; svelte-check 0 errors
fix(api,workers): envelope seam — accept real worker ambiguous/unscored shapes + docs + versions
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 1m1s
24c94d1a7e
Oracle final-gate fix: Core's strict validator rejected the worker's real
output for two shapes (ambiguous B carries the suggested target's score;
unscored B carries no identity lists) — any run with two similar people
would have been rejected wholesale and dead after 3 retries.

- WorkerResultService: ambiguous reviewOnly cards may carry exactly one
  consistent score (finite [0,2], clamp, 1e-5 tolerance); non-ambiguous
  reviewOnly must stay unscored; conflicting/compatiblePersonIds optional
  when absent for non-ambiguous reviewOnly; half-scored rejected
- ClassInstanceDiscoveryService: dead refsAllocatedThisJob removed,
  comment corrected (envelope validator + no-steal are the within-job
  guards)
- workers/people: semisupervised B confidence derived from the ROUNDED
  distance (exact clamp identity); pinning test
- Golden seam tests: byte-exact real-worker fixtures (ambiguous-with-score,
  unscored-no-lists, scored B, empty) through real submitResult on
  semisupervised jobs; 4 new PG tests + 6 new H2 validator tests
- Docs: ADR 0101 accepted (final contract + rollout constraint), spec 050
  T001/T002 done, ISSUE-087 mitigated
- Versions: web 0.0.40->0.0.41, api 1.21.62->1.21.63

Verified: 1481 API tests, 183 worker tests, svelte-check 0 errors;
independent oracle gate: APPROVED
chris merged commit 8401f99748 into main 2026-10-03 08:09:52 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!996
No description provided.