fix(people): embedding-only backfill to migrate legacy buffalo_l occurrences into arcface-r50 space #763

Merged
chris merged 1 commit from fix/legacy-embedding-space-backfill into main 2026-09-19 01:04:43 +02:00
Owner

Fixes the data half of ISSUE-049: accepted people face occurrences carrying legacy buffalo_l/arcface-v1 embeddings (pre-#702 worker stack) are invisible to cross-person comparison — matching, merge suggestions and similarity ranking group by embedding space and only pair within a shared space, so legacy-only clusters (115 of 116 active unnamed instances) can never be merge-suggested against named persons (all 21 named persons are arcface-r50-only).

What

Embedding-only backfill — NO face re-detection (stored bboxes: no bbox drift, no duplicate face rows):

  • Enqueue (WorkerJobEnqueueService.enqueueOwnerFaceEmbeddingBackfillJobs): one owner_face_embedding_backfill job per affected media item, payload faces array [{faceAnnotationId, boundingBox}...] capped at 500/job. Target query: accepted people face_annotation occurrences with a buffalo_l coe row and no arcface-r50/w600k-v1 row; photo-only; skips permanently_deleted media; face-level anti-join on active backfill jobs ⇒ repeated invocations drain the backlog resumably. Batching choice: per-media jobs instead of one owner-scoped job, because media download is authorized via per-job worker_media_access_grants created only from worker_jobs.media_item_id (owner-scoped jobs could not download pixels), and per-photo payloads are naturally bounded.
  • Trigger (POST /api/admin/people/backfill-legacy-embeddings, superadmin, returns {jobsEnqueued}). Deliberately NOT auto-scheduled.
  • Worker (main.py new dispatch branch): downloads the original once per job, crops each stored bbox via FaceDetector.embedding_from_normalized_box (same 10%-pad/112×112/[-1,1] preprocessing as detection time), stamps modelName/modelVersion from the same MEMORIES_FACE_EMBEDDING_MODEL_NAME/VERSION env vars as detection so the space label matches exactly, submits owner_face_embedding_backfill_result with per-face embeddings.
  • Result acceptance (WorkerResultService): routes the new result type through the existing upsertOccurrenceEmbedding path with explicit faceAnnotationId (V118 UNIQUE(occurrence_id, model_name, model_version) allows coexistence; dedup ⇒ idempotent). Faces whose occurrence is missing or no longer accepted are skipped + counted, never failing the result; superseded occurrences do NOT buffer pending rows (would re-arm recognition for settled faces). On acceptance, owner_face_recognition is enqueued for the owner (idempotent via the enqueue's own active-job guard) so matching/merge suggestions converge over the now-comparable snapshot.
  • Capability registration: V127 migration (discovery_domains.job_types + worker_credentials, V084/V111 pattern) + deploy/compose.yaml MEMORIES_WORKER_CAPABILITIES.
  • Cleanup recommendation (comment only, no code): legacy buffalo_l coe rows can be deleted once arcface coverage is verified.

Test plan

  • rtk ./gradlew :apps:api:compileJava green
  • New WorkerJobEnqueueServiceTest cases: buffalo-only occurrence selected, arcface-covered skipped, other domains skipped, active-job anti-join + resumable drain, photo/permanently_deleted filters, payload shape
  • New WorkerResultBackfillEmbeddingTest: arcface coe row written + recognition enqueued, idempotent replay, superseded/no-embedding skipped without failing, owner mismatch + wrong job type rejected
  • PeopleBackfillControllerTest: superadmin 403 + jobsEnqueued for the new endpoint
  • Full rtk ./gradlew :apps:api:test green
  • python -m py_compile on the worker; pytest in workers/people: 71 passed, 8 pre-existing failures unrelated to this change (reproduced with the worker changes stashed — numpy-version issues in mock-based detection tests)
  • No Flyway schema change for the backfill itself (V127 is capability registration only); KNOWN_ISSUES.md and specs/ untouched
Fixes the data half of ISSUE-049: accepted people face occurrences carrying legacy buffalo_l/arcface-v1 embeddings (pre-#702 worker stack) are invisible to cross-person comparison — matching, merge suggestions and similarity ranking group by embedding space and only pair within a shared space, so legacy-only clusters (115 of 116 active unnamed instances) can never be merge-suggested against named persons (all 21 named persons are arcface-r50-only). ## What Embedding-only backfill — NO face re-detection (stored bboxes: no bbox drift, no duplicate face rows): - **Enqueue** (`WorkerJobEnqueueService.enqueueOwnerFaceEmbeddingBackfillJobs`): one `owner_face_embedding_backfill` job per affected media item, payload `faces` array [{faceAnnotationId, boundingBox}...] capped at 500/job. Target query: accepted people face_annotation occurrences with a buffalo_l coe row and no arcface-r50/w600k-v1 row; photo-only; skips permanently_deleted media; face-level anti-join on active backfill jobs ⇒ repeated invocations drain the backlog resumably. **Batching choice:** per-media jobs instead of one owner-scoped job, because media download is authorized via per-job `worker_media_access_grants` created only from `worker_jobs.media_item_id` (owner-scoped jobs could not download pixels), and per-photo payloads are naturally bounded. - **Trigger** (`POST /api/admin/people/backfill-legacy-embeddings`, superadmin, returns {jobsEnqueued}). Deliberately NOT auto-scheduled. - **Worker** (`main.py` new dispatch branch): downloads the original once per job, crops each stored bbox via `FaceDetector.embedding_from_normalized_box` (same 10%-pad/112×112/[-1,1] preprocessing as detection time), stamps modelName/modelVersion from the same `MEMORIES_FACE_EMBEDDING_MODEL_NAME/VERSION` env vars as detection so the space label matches exactly, submits `owner_face_embedding_backfill_result` with per-face embeddings. - **Result acceptance** (`WorkerResultService`): routes the new result type through the existing `upsertOccurrenceEmbedding` path with explicit faceAnnotationId (V118 UNIQUE(occurrence_id, model_name, model_version) allows coexistence; dedup ⇒ idempotent). Faces whose occurrence is missing or no longer accepted are skipped + counted, never failing the result; superseded occurrences do NOT buffer pending rows (would re-arm recognition for settled faces). On acceptance, `owner_face_recognition` is enqueued for the owner (idempotent via the enqueue's own active-job guard) so matching/merge suggestions converge over the now-comparable snapshot. - **Capability registration**: V127 migration (discovery_domains.job_types + worker_credentials, V084/V111 pattern) + deploy/compose.yaml MEMORIES_WORKER_CAPABILITIES. - **Cleanup recommendation (comment only, no code)**: legacy buffalo_l coe rows can be deleted once arcface coverage is verified. ## Test plan - [x] `rtk ./gradlew :apps:api:compileJava` green - [x] New `WorkerJobEnqueueServiceTest` cases: buffalo-only occurrence selected, arcface-covered skipped, other domains skipped, active-job anti-join + resumable drain, photo/permanently_deleted filters, payload shape - [x] New `WorkerResultBackfillEmbeddingTest`: arcface coe row written + recognition enqueued, idempotent replay, superseded/no-embedding skipped without failing, owner mismatch + wrong job type rejected - [x] `PeopleBackfillControllerTest`: superadmin 403 + jobsEnqueued for the new endpoint - [x] Full `rtk ./gradlew :apps:api:test` green - [x] `python -m py_compile` on the worker; pytest in workers/people: 71 passed, 8 pre-existing failures unrelated to this change (reproduced with the worker changes stashed — numpy-version issues in mock-based detection tests) - [x] No Flyway schema change for the backfill itself (V127 is capability registration only); KNOWN_ISSUES.md and specs/ untouched
fix(people): embedding-only backfill to migrate legacy buffalo_l occurrences into arcface-r50 space
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 1m45s
b2a6858d85
Accepted people face occurrences carrying legacy buffalo_l/arcface-v1
class_occurrence_embeddings (pre-PR#702 worker-ml stack) are invisible to
all cross-person comparison: matching, merge suggestions and similarity
ranking group by embedding space and only pair within a shared space, so
legacy-only clusters can never be merge-suggested against named persons
(ISSUE-049).

Embedding-only backfill (no re-detection — stored bboxes, no bbox drift,
no duplicate face rows):

- WorkerJobEnqueueService.enqueueOwnerFaceEmbeddingBackfillJobs(): one
  owner_face_embedding_backfill job per affected media item (faces array
  with stored bboxes, cap 500/job). Media-scoped batching chosen because
  media download is authorized via per-job worker_media_access_grants
  created only from worker_jobs.media_item_id — an owner-scoped job could
  not download pixels — and per-photo payloads are naturally bounded.
  Target query: accepted people face_annotation occurrences with a
  buffalo_l row and no arcface-r50/w600k-v1 row, photo-only, skipping
  permanently_deleted media, with a face-level anti-join on active
  backfill jobs so repeated endpoint invocations drain the backlog.
- PeopleBackfillController POST /api/admin/people/backfill-legacy-
  embeddings (superadmin, {jobsEnqueued}). Not auto-scheduled.
- Worker: new dispatch branch downloads the original once per job, crops
  each stored bbox with the detection-time preprocessing
  (FaceDetector.embedding_from_normalized_box) and stamps
  modelName/modelVersion from the same MEMORIES_FACE_EMBEDDING_MODEL_*
  env vars as detection, so the space label matches exactly.
- WorkerResultService.acceptOwnerFaceEmbeddingBackfillResult(): writes
  coe rows via the existing upsertOccurrenceEmbedding path (dedup =>
  idempotent); faces whose occurrence is missing or no longer accepted
  are skipped + counted, never failing the result (no pending buffer for
  superseded occurrences — that would re-arm recognition for settled
  faces). On acceptance, owner_face_recognition is enqueued for the
  owner (idempotent via the enqueue's own active-job guard) so matching
  and merge suggestions converge over the now-comparable snapshot.
- V127 migration: register the job type for the people worker credential
  (discovery_domains.job_types + worker_credentials allowed_job_types,
  V084/V111 pattern); compose capabilities updated.
- Follow-up (comment only): legacy buffalo_l rows can be deleted once
  arcface coverage is verified.

Tests: enqueue selection (buffalo-only selected, arcface-covered/plants
domain/other-owner skipped, active-job anti-join + resumable drain,
photo/permanently-deleted filters); result acceptance (arcface row
written + recognition enqueued, idempotent replay, superseded/no-
embedding skipped without failing, owner mismatch + wrong job type
rejected); controller superadmin checks.
chris merged commit 3c4c9966fb into main 2026-09-19 01:04:43 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!763
No description provided.