fix(workers): cap people worker ONNX/BLAS thread pools to reduce CPU load #721

Merged
chris merged 2 commits from fix/people-worker-thread-cap into main 2026-09-15 22:46:41 +02:00
Owner

Problem

The people worker (single Python process, memories-worker-people) consumes ~430-570% CPU on the host, saturating the box.

Root cause

Nothing bounds its thread pools:

  1. ONNX Runtime: both sessions (RetinaFace detection det_10g.onnx, ArcFace embedding w600k_r50.onnx) were created with default SessionOptions, so each session spawned an intra-op thread pool of all cores (workers/people/src/memories_people_worker/face_model.py).
  2. OpenBLAS: the float64 matmuls in the owner_face_recognition clustering jobs (main.py, numpy) made OpenBLAS default to one thread per core, since no OMP/BLAS env vars were set.

Fix (matches the plants worker pattern)

  • face_model.py: both InferenceSessions now use an explicit ort.SessionOptions() with intra_op_num_threads = 2 / inter_op_num_threads = 1 (same as workers/plants model.py), plus explicit providers=["CPUExecutionProvider"] (same as workers/ml). Comment notes the caps must match the compose env.
  • deploy/compose.yaml: memories-worker-people environment now pins OMP_NUM_THREADS=2, OPENBLAS_NUM_THREADS=2, MKL_NUM_THREADS=2 to cap BLAS threads used by numpy.
  • deploy/example.env: documents the three vars with a one-line comment.

Verification

  • python3.12 -m compileall on face_model.py: OK; import + AST check confirms SessionOptions, thread caps, and CPUExecutionProvider are present.
  • People worker test suite (PYTHONPATH=src python3.12 -m pytest tests/): 43 passed, 8 failed - the same 8 tests in test_face_model.py fail on clean origin/main (verified via git stash re-run), i.e. pre-existing failures from outdated mocks vs. the current flat-output FPN decoding, unrelated to this change.
  • docker compose -f deploy/compose.yaml --env-file deploy/example.env config --quiet: valid; rendered config shows the three thread-cap vars on memories-worker-people.
  • Diff touches exactly the three intended files; no behavior change beyond the caps.

Part 1 of 2 of the CPU load fix (BLAS/OpenMP caps in the API still to come).

## Problem The people worker (single Python process, `memories-worker-people`) consumes ~430-570% CPU on the host, saturating the box. ## Root cause Nothing bounds its thread pools: 1. **ONNX Runtime**: both sessions (RetinaFace detection `det_10g.onnx`, ArcFace embedding `w600k_r50.onnx`) were created with default `SessionOptions`, so each session spawned an intra-op thread pool of *all* cores (`workers/people/src/memories_people_worker/face_model.py`). 2. **OpenBLAS**: the float64 matmuls in the `owner_face_recognition` clustering jobs (`main.py`, numpy) made OpenBLAS default to one thread per core, since no `OMP/BLAS` env vars were set. ## Fix (matches the plants worker pattern) - `face_model.py`: both `InferenceSession`s now use an explicit `ort.SessionOptions()` with `intra_op_num_threads = 2` / `inter_op_num_threads = 1` (same as `workers/plants` `model.py`), plus explicit `providers=["CPUExecutionProvider"]` (same as `workers/ml`). Comment notes the caps must match the compose env. - `deploy/compose.yaml`: `memories-worker-people` environment now pins `OMP_NUM_THREADS=2`, `OPENBLAS_NUM_THREADS=2`, `MKL_NUM_THREADS=2` to cap BLAS threads used by numpy. - `deploy/example.env`: documents the three vars with a one-line comment. ## Verification - `python3.12 -m compileall` on `face_model.py`: OK; import + AST check confirms `SessionOptions`, thread caps, and `CPUExecutionProvider` are present. - People worker test suite (`PYTHONPATH=src python3.12 -m pytest tests/`): 43 passed, 8 failed - the same 8 tests in `test_face_model.py` fail on clean `origin/main` (verified via `git stash` re-run), i.e. pre-existing failures from outdated mocks vs. the current flat-output FPN decoding, unrelated to this change. - `docker compose -f deploy/compose.yaml --env-file deploy/example.env config --quiet`: valid; rendered config shows the three thread-cap vars on `memories-worker-people`. - Diff touches exactly the three intended files; no behavior change beyond the caps. Part 1 of 2 of the CPU load fix (BLAS/OpenMP caps in the API still to come).
fix(workers): cap people worker ONNX/BLAS thread pools to reduce CPU load
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 4s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 3m11s
d48ef27d99
The people worker consumed ~430-570% CPU because nothing bounded its
thread pools:

- ONNX Runtime sessions (detection + embedding) used default
  SessionOptions, giving each session an intra-op pool of all cores.
  Now created with explicit SessionOptions: intra_op_num_threads=2,
  inter_op_num_threads=1, providers=[CPUExecutionProvider], mirroring
  the plants worker pattern.
- OpenBLAS in numpy (owner_face_recognition clustering) defaulted to
  all cores. Compose now pins OMP_NUM_THREADS/OPENBLAS_NUM_THREADS/
  MKL_NUM_THREADS=2 on memories-worker-people and documents the vars
  in deploy/example.env.
fix(deploy): scope BLAS caps to people service; bump version
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 5s
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 3m2s
dba1620269
Review follow-up on PR #721:

- deploy/example.env: comment out OMP/OPENBLAS/MKL_NUM_THREADS. The file
  is loaded via env_file by all compose services, so active values there
  would cap BLAS stack-wide (including workers that did not opt in).
  Comment now states this and points at the per-service pinning in
  deploy/compose.yaml, which remains the sole source of the caps.
- gradle.properties: bump memoriesVersion 1.21.30 -> 1.21.31 (in-branch
  bump convention for runtime PRs).
chris merged commit 50ee4ea8e9 into main 2026-09-15 22:46:41 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!721
No description provided.