feat(workers): V6 fingerprint-keyed representative cache (ADR 0113) #1269

Merged
chris merged 2 commits from feat/rep-cache-worker into main 2026-10-16 21:29:15 +02:00
Owner

Summary

Worker side of ADR 0113 V6 (fingerprint-keyed representative cache) — the biggest remaining per-run cost after the v3 graph optimization: _person_representatives was recomputed from scratch over the whole labeled library every run (~1–2 min at owner scale, pure deterministic waste on unchanged inputs).

  • clustering.py: _representative_fingerprint(rows, split, model) — the pinned digest (sha256 over sorted pair-local usable face ids + :.3f split + model pair); _decompose_person_rows — the ADR 0108 per-person body extracted verbatim; _person_representatives_with_fingerprints(embeddings, assignments, split, *, persisted=None) — row selection performed ONCE, fingerprint computed inside that path (zero selection duplication), per person: fingerprint-equal persisted entry with non-empty reps → reuse verbatim (skips even unit_rows, not just linkage); else recompute. _person_representatives stays a thin full-recompute wrapper (signature/behavior unchanged).
  • core_api.py: fetch_owner_face_embeddings returns a typed OwnerFaceFeed(embeddings, person_representatives) reading the new top-level response key (missing/non-list → [] = pure recompute for older Core).
  • main.py: handler applies the cache method-independently (v2 AND v3) before run_clustering; _representative_cache_from_entries treats every malformed row — including explicit JSON null fingerprints from the API response — as a cache MISS, never an error; _person_representatives_payload gains the additive optional fingerprint so the envelope full-replaces with the reused-or-recomputed set + its digest.

Test plan

  • New test_representative_fingerprint_cache.py (15 tests): digest material + determinism/permutation/split/model-pair sensitivity, the V6 sub-gate (monkeypatch counter: zero cosine_distance_matrix calls on hits, full recompute on miss), byte-identical hit parity through the envelope JSON round trip, handler-level hit/stale/v3-independence/empty-cache.
  • 11 handler stubs updated mechanically for the typed feed; test_multi_representant.py pins every shipped entry carries a non-empty fingerprint.
  • Full worker suite: 378 passed, 1 skipped (pre-existing heif), 99 subtests — v1/v2/v3 + registry + prototype + rep_graph + golden all green.
  • Cross-lane contract verified against the API branch feat/rep-cache-api: entry keys, 64-hex definition, null-as-miss. Merge order: API PR first, then this one.
## Summary Worker side of ADR 0113 V6 (fingerprint-keyed representative cache) — the biggest remaining per-run cost after the v3 graph optimization: `_person_representatives` was recomputed from scratch over the whole labeled library every run (~1–2 min at owner scale, pure deterministic waste on unchanged inputs). - **`clustering.py`**: `_representative_fingerprint(rows, split, model)` — the pinned digest (sha256 over sorted pair-local usable face ids + `:.3f` split + model pair); `_decompose_person_rows` — the ADR 0108 per-person body extracted verbatim; `_person_representatives_with_fingerprints(embeddings, assignments, split, *, persisted=None)` — row selection performed ONCE, fingerprint computed inside that path (zero selection duplication), per person: fingerprint-equal persisted entry with non-empty reps → reuse verbatim (skips even `unit_rows`, not just linkage); else recompute. `_person_representatives` stays a thin full-recompute wrapper (signature/behavior unchanged). - **`core_api.py`**: `fetch_owner_face_embeddings` returns a typed `OwnerFaceFeed(embeddings, person_representatives)` reading the new top-level response key (missing/non-list → `[]` = pure recompute for older Core). - **`main.py`**: handler applies the cache method-independently (v2 AND v3) before `run_clustering`; `_representative_cache_from_entries` treats every malformed row — including explicit JSON `null` fingerprints from the API response — as a cache MISS, never an error; `_person_representatives_payload` gains the additive optional `fingerprint` so the envelope full-replaces with the reused-or-recomputed set + its digest. ## Test plan - New `test_representative_fingerprint_cache.py` (15 tests): digest material + determinism/permutation/split/model-pair sensitivity, the V6 sub-gate (monkeypatch counter: zero `cosine_distance_matrix` calls on hits, full recompute on miss), byte-identical hit parity through the envelope JSON round trip, handler-level hit/stale/v3-independence/empty-cache. - 11 handler stubs updated mechanically for the typed feed; `test_multi_representant.py` pins every shipped entry carries a non-empty fingerprint. - Full worker suite: **378 passed, 1 skipped (pre-existing heif), 99 subtests** — v1/v2/v3 + registry + prototype + rep_graph + golden all green. - Cross-lane contract verified against the API branch `feat/rep-cache-api`: entry keys, 64-hex definition, null-as-miss. Merge order: API PR first, then this one.
_person_representatives_with_fingerprints performs the (model pair, person)
row selection ONCE, computes the pinned sha256 fingerprint over the exact
split inputs (sorted usable face ids + split:.3f + modelName + modelVersion),
and per person either reuses a fingerprint-matched persisted decomposition
verbatim (no linkage recompute) or recomputes via _decompose_person_rows
(the unchanged ADR 0108 body). _person_representatives stays a thin
full-recompute wrapper — existing call sites and suites byte-identical.

Tests: fingerprint determinism/sensitivity + pinned digest material,
hit-skips-cosine_distance_matrix sub-gate, stale/empty-cache misses.
feat(worker): ADR 0113 V6 — wire the fingerprint-keyed representative cache through handler + client
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
6933ef5b86
core_api: fetch_owner_face_embeddings now returns OwnerFaceFeed(embeddings,
person_representatives) — the API's new top-level personRepresentatives key
(envelope shape + fingerprint) rides the same response; missing key (older
Core) degrades to an empty list = pure recompute.

main: the v2/v3 representative build calls
_person_representatives_with_fingerprints with the parsed persisted set
(_representative_cache_from_entries; malformed rows drop to cache misses);
_person_representatives_payload gains the additive optional fingerprint
field per entry, so the envelope always ships the reused-or-recomputed set
with its cache key for the API full-replace.

Tests: byte-identical hit parity through the envelope round trip,
handler-level hit (no cosine_distance_matrix call, same fingerprints
shipped) / stale-fingerprint miss (recompute + new fingerprint), v3
method-independence, empty-cache passthrough; 11 handler stubs updated to
OwnerFaceFeed; multi-representant envelope test pins fingerprint parity.
chris merged commit 6641474990 into main 2026-10-16 21:29:15 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1269
No description provided.