fix(people): face pipeline — B1 remove unsound rekey + B2 locking/insert-only acceptance #718

Merged
chris merged 4 commits from fix/face-pipeline-embedding-flush into main 2026-09-15 17:30:51 +02:00
Owner

Scope fix (replacement head: fix/face-pipeline-b1b2-locks @ e25b72f54194556a36ab6357a87a73e5af798a87)

Replaces the original head fix/face-pipeline-embedding-flush @ 1d7d8e14 per the review-718 integrity adjudication.Worker-side vectorized clustering from the replaced head is preserved on the frozen worktree and lands separately.

B1 — embedding-flush scheduler — removal rationale

PendingEmbeddingReconciliationScheduler.rekeyLegacyFaceOccurrences is REMOVED, not fixed. The "exactly one tentative face on the media" heuristic is unsound for any photo with 2+ faces: several fallback occurrences raced the same unique constraint (DuplicateKeyException, observed live) and a guess would silently move owner data. Scope of this PR is future-data repair only; existing fallback-keyed occurrences are intentionally left as-is.

B2 — automated acceptance is locking + insert-only

  • letFacesForUpdate: deterministic sorted SELECT … FOR UPDATE on the pre-existing discovery_results face rows (lock order: face rows → class_instances → occurrences).
  • insertOccurrenceNoSteal: ON CONFLICT (domain_key, owner_id, occurrence_type, occurrence_key) DO NOTHING RETURNING id (H2 MERGE KEY fixture fallback). Automated paths NEVER update/reassign an already accepted occurrence.
  • M1 format: auto-create gated to new_person_cluster; mixed clusters d�eduplicate per-face via filterAssignedFaceRefs; a candidate whose faces are all accepted elsewhere is decided rejected (lost_insert_race / all_faces_already_assigned) WITHOUT creating an empty person.
  • M2 policy auto-accept: locked, allowMove=false, skipped_assigned_refs skip note.
  • M3/M4 (manual owner decisions): lock first; human decisions keep allowMove=true.
  • M5 ClassInstanceService.upsertGenericPeopleOccurrence takes a single-face FOR UPDATE lock; regression-scope callers verified @Transactional, legacy retrospective entry point annotated.

Evidence

Test Assertions Result
Pr718IntegrityRegressionTest (7 cases incl. 2 b1/5 b2) no-steal ownership, no-empty-person, insert-only race convergence (1 person / 1 occurrence), non-candidate-type still reviewable, scheduler survives fallback flush 7/7 GREEN @ e25b72f5
PendingEmbeddingReconciliationSchedulerTest fallback occurrence untouched + pending evidence retained + purge tests green GREEN
:apps:api:test full suite 0 failures GREEN
:apps:worker-media:test 0 failures GREEN

Recovery limitations (honest)

Existing data is NOT fully resolved by this PR:

  • Fallback-keyed accepted occurrences (349 occurrence-less person instances observed on owner DB) remain unless a follow-up recovery endpoint is built.
  • Proposed follow-up: exact-ID repair driven by accepted candidates' occurrence_refs / representative_evidence / existing_instance_id — no fuzzy matching.
  • Worker people-clustering RAM ceiling ~40k faces (O(n²) distance matrix) remains mitigated, ceiling documented in KNOWN_ISSUES.
## Scope fix (replacement head: `fix/face-pipeline-b1b2-locks` @ `e25b72f54194556a36ab6357a87a73e5af798a87`) Replaces the original head `fix/face-pipeline-embedding-flush` @ `1d7d8e14` per the review-718 integrity adjudication.Worker-side vectorized clustering from the replaced head is preserved on the frozen worktree and lands separately. ## B1 — embedding-flush scheduler — removal rationale `PendingEmbeddingReconciliationScheduler.rekeyLegacyFaceOccurrences` is REMOVED, not fixed. The "exactly one tentative face on the media" heuristic is unsound for any photo with 2+ faces: several fallback occurrences raced the same unique constraint (`DuplicateKeyException`, observed live) and a guess would silently move owner data. Scope of this PR is future-data repair only; existing fallback-keyed occurrences are intentionally left as-is. ## B2 — automated acceptance is locking + insert-only - `letFacesForUpdate`: deterministic sorted `SELECT … FOR UPDATE` on the pre-existing `discovery_results` face rows (lock order: face rows → class_instances → occurrences). - `insertOccurrenceNoSteal`: `ON CONFLICT (domain_key, owner_id, occurrence_type, occurrence_key) DO NOTHING RETURNING id` (H2 `MERGE KEY` fixture fallback). Automated paths NEVER update/reassign an already accepted occurrence. - M1 format: auto-create gated to `new_person_cluster`; mixed clusters d�eduplicate per-face via `filterAssignedFaceRefs`; a candidate whose faces are all accepted elsewhere is decided `rejected` (`lost_insert_race` / `all_faces_already_assigned`) WITHOUT creating an empty person. - M2 policy auto-accept: locked, `allowMove=false`, `skipped_assigned_refs` skip note. - M3/M4 (manual owner decisions): lock first; human decisions keep `allowMove=true`. - M5 `ClassInstanceService.upsertGenericPeopleOccurrence` takes a single-face FOR UPDATE lock; regression-scope callers verified `@Transactional`, legacy retrospective entry point annotated. ## Evidence | Test | Assertions | Result | |---|---|---| | `Pr718IntegrityRegressionTest` (7 cases incl. 2 b1/5 b2) | no-steal ownership, no-empty-person, insert-only race convergence (1 person / 1 occurrence), non-candidate-type still reviewable, scheduler survives fallback flush | 7/7 GREEN @ `e25b72f5` | | `PendingEmbeddingReconciliationSchedulerTest` | fallback occurrence untouched + pending evidence retained + purge tests green | GREEN | | `:apps:api:test` full suite | 0 failures | GREEN | | `:apps:worker-media:test` | 0 failures | GREEN | ## Recovery limitations (honest) Existing data is NOT fully resolved by this PR: - Fallback-keyed accepted occurrences (349 occurrence-less person instances observed on owner DB) remain unless a follow-up recovery endpoint is built. - Proposed follow-up: exact-ID repair driven by accepted candidates' `occurrence_refs` / `representative_evidence` / `existing_instance_id` — no fuzzy matching. - Worker people-clustering RAM ceiling ~40k faces (O(n²) distance matrix) remains `mitigated`, ceiling documented in KNOWN_ISSUES.
fix(people): repair face pipeline — occurrence keys, embedding flush, loop guard, vectorized clustering
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 5m29s
1d7d8e142f
Root cause 1 — occurrence type/key mismatch killed embedding flush:
- discovery_people candidate refs carry only mediaItemId; occurrences were stored
  with fallback keys (mediaItemId:index) and type generic_evidence, while every
  embedding-write path filters occurrence_type='face_annotation' AND
  occurrence_key=<faceAnnotationId>. Production: 16,411 buffered pending
  embeddings, 24 flushed.
- ClassInstanceDiscoveryService.occurrenceType() now derives face_annotation
  when the ref resolves to a people-domain discovery_results face (even when the
  worker omitted type); the flush event (OccurrenceCreatedEvent) also fires on
  the UPDATE/relink path via publishOccurrenceAccepted.
- PendingEmbeddingReconciliationScheduler: primary flush requires
  occurrence_type='face_annotation'; adds idempotent legacy re-key pass for
  unambiguous fallback-keyed occurrences (single tentative face per media with
  pending embedding) then flushes again — drains the stuck backlog without a
  risky data migration.

Root cause 2 — people page empty:
- persistWorkerCandidates auto-create passed the in-memory candidate (camelCase
  occurrenceRefs) to acceptCandidateOccurrences which reads the DB column name
  (occurrence_refs) → 0 occurrences → instances() (requires accepted
  occurrence) returned nothing. Now re-reads the persisted row; auto-create
  extended to all unmatched people/person candidates (existing_person_match
  without target previously stuck pending forever).

Root cause 3 — infinite recognition loop:
- 0-candidate owner_face_recognition results left faces owner_tentative while
  the post-completion listener and the 5-min scheduler re-enqueued endlessly.
- New FaceRecognitionProgressService computes unconsumed-embedding evidence;
  both triggers only arm a job when tentative faces hold pending or unassigned
  accepted embeddings. New evidence (new pending rows) re-arms recognition.

Root cause 4 — O(n^2) pure-Python clustering:
- _density_clusters / _best_existing_person_match vectorized with NumPy
  pairwise cosine distance matrices (numpy already a dependency). Golden tests
  (workers/people/tests/test_clustering_golden.py) prove identical output vs
  the naive reference implementation incl. chain expansion order, zero-norm
  sentinels, tie-breaks; scalar _cosine_distance call count asserted 0.

Tests: full :apps:api test suite green (Testcontainers PG + H2 fixtures);
people worker pytest green for clustering/core/claim tests (pre-existing
face_model ONNX failures unchanged).
chris changed title from fix(people): face pipeline — occurrence keys/embedding flush, loop guard, vectorized clustering to fix(people): face pipeline — B1 remove unsound rekey + B2 locking/insert-only acceptance 2026-09-15 17:02:27 +02:00
fix(people): B1/B2 face-occurrence integrity — remove unsound rekey, sorted face-row locking, insert-only auto acceptance
All checks were successful
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 2m58s
Build and Deploy / Detect changed services and test scopes (push) Successful in 5s
e25b72f541
B1: PendingEmbeddingReconciliationScheduler no longer re-keys legacy
fallback-keyed occurrences ("mediaItemId:index") — the single-face
heuristic is unsound for multi-face media (DuplicateKeyException +
permanently-wrong assignments on the owner DB). Fallback-keyed rows stay
untouched; repair moves to a future exact-ID admin recovery endpoint.

B2: ClassInstanceDiscoveryService acceptance is now locking + insert-only
for automated paths (review-718 invariant I1/I2):
- letFacesForUpdate: deterministic SELECT ... FOR UPDATE on the
  pre-existing people discovery_results face rows (lock order: face rows
  -> class_instances -> occurrences)
- insertOccurrenceNoSteal: ON CONFLICT DO NOTHING RETURNING (H2 MERGE
  fallback), auto path never updates/reassigns an accepted occurrence
- M1 auto-create gated to new_person_cluster; mixed cluster dedupes via
  filterAssignedFaceRefs; all-assigned candidate decided rejected
  (lost_insert_race) without an empty instance
- M2 policy auto-accept locked + allowMove=false + skip decision
- M3/M4 manual owner decisions lock first and keep allowMove=true
- M5 upsertGenericPeopleOccurrence takes single-face FOR UPDATE;
  retrospective entry point annotated @Transactional

Tests: ported 7 red Pr718IntegrityRegressionTest cases (now green,
sen-1-approved helper semantics: counted non-empty persons + direct
no-steal ownership asserts) + scheduler manifest seed; legacy
auto-create fixtures aligned to insert-only semantics; version 1.21.29.
docs(known-issues): ISSUE-031 recovery-deferred + ISSUE-032 memory ceiling mitigated [PR#718]
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 6s
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
a5833edabb
chris merged commit eef360d107 into main 2026-09-15 17:30:51 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!718
No description provided.