fix(workers): cap people worker ONNX/BLAS thread pools to reduce CPU load #721
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/people-worker-thread-cap"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
The people worker (single Python process,
memories-worker-people) consumes ~430-570% CPU on the host, saturating the box.Root cause
Nothing bounds its thread pools:
det_10g.onnx, ArcFace embeddingw600k_r50.onnx) were created with defaultSessionOptions, so each session spawned an intra-op thread pool of all cores (workers/people/src/memories_people_worker/face_model.py).owner_face_recognitionclustering jobs (main.py, numpy) made OpenBLAS default to one thread per core, since noOMP/BLASenv vars were set.Fix (matches the plants worker pattern)
face_model.py: bothInferenceSessions now use an explicitort.SessionOptions()withintra_op_num_threads = 2/inter_op_num_threads = 1(same asworkers/plantsmodel.py), plus explicitproviders=["CPUExecutionProvider"](same asworkers/ml). Comment notes the caps must match the compose env.deploy/compose.yaml:memories-worker-peopleenvironment now pinsOMP_NUM_THREADS=2,OPENBLAS_NUM_THREADS=2,MKL_NUM_THREADS=2to cap BLAS threads used by numpy.deploy/example.env: documents the three vars with a one-line comment.Verification
python3.12 -m compileallonface_model.py: OK; import + AST check confirmsSessionOptions, thread caps, andCPUExecutionProviderare present.PYTHONPATH=src python3.12 -m pytest tests/): 43 passed, 8 failed - the same 8 tests intest_face_model.pyfail on cleanorigin/main(verified viagit stashre-run), i.e. pre-existing failures from outdated mocks vs. the current flat-output FPN decoding, unrelated to this change.docker compose -f deploy/compose.yaml --env-file deploy/example.env config --quiet: valid; rendered config shows the three thread-cap vars onmemories-worker-people.Part 1 of 2 of the CPU load fix (BLAS/OpenMP caps in the API still to come).