fix(workers/people): blockwise float32 density graph (OOM segfault at 26k faces) #975

Merged
chris merged 1 commit from fix/constrained-clustering-oom into main 2026-10-01 09:33:33 +02:00
Owner

Summary

The constrained forward clustering pass (ADR 0100 / PR #969) builds its density graph over anchor + all unassigned faces of an owner. On a household-scale library (~26k faces in one model space) that meant a full N x N float64 similarity matrix plus two full-size temporaries: ~20.7 GiB simultaneous allocations, which segfaulted OpenBLAS mid-GEMM on the 16 GB deploy host. Every owner_face_recognition job since deploy died hard after fetching embeddings (kernel: segfault ... in libscipy_openblas64_), failed after 3 attempts, and submitted nothing — so the suggestion carousel only ever showed new_person_cluster entries.

Fix

_density_graph is now blockwise float32 with a bounded ~192 MiB tile:

  • pairwise similarities computed in row blocks (peak scratch = block x n), never a full N x N matrix
  • cannot-link suppression via label-broadcast comparison per block (replaces the O(n²) Python double loop — minutes of CPU even without the crash)
  • in-place mask handling (nan_to_num(copy=False), no np.triu copy)
  • legacy _cosine_distance_matrix gets the same float32 blockwise treatment

Measured at n=26213, dim=512: graph build 12 s, peak RSS 2.4 GiB (was: segfault), adjacency lists byte-identical to the brute-force reference across labeled / zero-norm / dimension-mismatch / block-boundary shapes.

Tests

  • brute-force equivalence pins (with/without labels, degenerate rows, n around block boundaries)
  • household-scale memory pin: 26k nodes build under 1 GiB traced allocations
  • all 124 worker tests green (incl. existing golden + constrained suites)

Post-merge

After deploy, the failed owner_face_recognition job needs a re-enqueue (retry or new dirty face) so the carousel fills with the pending merge suggestions for the confirmed anchors.

## Summary The constrained forward clustering pass (ADR 0100 / PR #969) builds its density graph over **anchor + all unassigned faces** of an owner. On a household-scale library (~26k faces in one model space) that meant a full N x N **float64** similarity matrix plus two full-size temporaries: **~20.7 GiB simultaneous allocations**, which segfaulted OpenBLAS mid-GEMM on the 16 GB deploy host. Every `owner_face_recognition` job since deploy died hard after fetching embeddings (kernel: `segfault ... in libscipy_openblas64_`), failed after 3 attempts, and submitted nothing — so the suggestion carousel only ever showed `new_person_cluster` entries. ## Fix `_density_graph` is now **blockwise float32** with a bounded ~192 MiB tile: - pairwise similarities computed in row blocks (peak scratch = block x n), never a full N x N matrix - cannot-link suppression via label-broadcast comparison per block (replaces the O(n²) Python double loop — minutes of CPU even without the crash) - in-place mask handling (`nan_to_num(copy=False)`, no `np.triu` copy) - legacy `_cosine_distance_matrix` gets the same float32 blockwise treatment Measured at n=26213, dim=512: graph build **12 s**, peak RSS **2.4 GiB** (was: segfault), adjacency lists byte-identical to the brute-force reference across labeled / zero-norm / dimension-mismatch / block-boundary shapes. ## Tests - brute-force equivalence pins (with/without labels, degenerate rows, n around block boundaries) - household-scale memory pin: 26k nodes build under 1 GiB traced allocations - all 124 worker tests green (incl. existing golden + constrained suites) ## Post-merge After deploy, the failed `owner_face_recognition` job needs a re-enqueue (retry or new dirty face) so the carousel fills with the pending merge suggestions for the confirmed anchors.
fix(workers/people): blockwise float32 density graph — OOM/segfault at 26k faces
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
8834f71a8f
The constrained forward pass (ADR 0017) built its density graph over
anchors + all unassigned faces in one shot: a full N x N float64
similarity matrix plus two full-size temporaries (~20.7 GiB simultaneous
allocations at n=26k, dim=512). On the 16 GB deploy host this segfaulted
OpenBLAS mid-GEMM (kernel log: 'segfault ... in
libscipy_openblas64_' at every owner_face_recognition attempt), the
worker died before submitting results, and every job failed after 3
attempts. With no successful recognition run since the anchors were
confirmed, the suggestion carousel only ever showed new-person clusters.

The graph build is now blockwise float32 with a bounded ~192 MiB tile,
vectorized label-broadcast cannot-link (replacing the O(n^2) Python
double loop that would have taken minutes even without the crash), and
in-place mask handling. The legacy _cosine_distance_matrix gets the same
float32 blockwise treatment. Measured at n=26213: peak RSS 2.4 GiB
(down from ~20.7 GiB logical allocation), graph build 12 s, adjacency
lists byte-identical to the brute-force reference across labeled,
degenerate (zero-norm / dimension-mismatch), and block-boundary shapes.

New regression pins: brute-force equivalence (with/without labels),
household-scale memory bound (26k nodes < 1 GiB traced), plus the
existing 121 worker tests all green.
chris merged commit e7d88aa837 into main 2026-10-01 09:33:33 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!975
No description provided.