chore(api): cap JVM heap at 6% of host via JAVA_TOOL_OPTIONS #1246

Merged
chris merged 1 commit from chore/api-heap-cap into main 2026-10-15 22:41:50 +02:00
Owner

Problem

Production memories-api RSS measured 4.68 GiB (2026-10-15) on a 46 GiB host. Oracle diagnosis (ora-1): not a leak — G1 high-water mark:

  • The api service has no cgroup memory limit and no explicit -Xmx, so the JVM default MaxRAMPercentage=25 applies to the host → max heap ≈ 11.7 GiB.
  • /proc/1/maps shows an 11,536 MB PROT_NONE G1 reservation (JRE-only image, no jcmd).
  • Recognition-result storms (~20 MB owner_face_recognition POSTs with 512-dim boxed embeddings + full-pool embeddings GET) push G1 to ~4 GiB committed, which it never uncommits (G1PeriodicGC default off) → RSS ratchets across container lifetime: 408 MB @5min fresh → 1.15 GiB @35min baseline → 4.68 GiB after 3 recognition transients in one evening.
  • Code audit found zero unbounded retention structures (SSE broadcaster self-cleans, no Caffeine/Guava caches; multipart spools to disk).
  • Tail risk: unbounded heap means a future loop regression races the host OOM-killer (JVM = fattest RSS).

Fix

One env line in deploy/compose.yaml: JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=6 -XX:InitialRAMPercentage=2 (≈2.8 GiB max, overridable via MEMORIES_API_JAVA_TOOL_OPTIONS).

Out of scope (deliberately)

  • G1PeriodicGCInterval uncommit tuning (aesthetic; rec #1 bounds RSS already)
  • Streaming the 20 MB result POST / paging the embeddings GET — YAGNI at 2–4 runs/day; revisit at ~10× pool size
  • Actuator jvm_memory_used metrics exposure — separate small change if wanted
## Problem Production `memories-api` RSS measured **4.68 GiB** (2026-10-15) on a 46 GiB host. Oracle diagnosis (ora-1): **not a leak** — G1 high-water mark: - The api service has **no cgroup memory limit** and no explicit `-Xmx`, so the JVM default `MaxRAMPercentage=25` applies to the **host** → max heap ≈ 11.7 GiB. - `/proc/1/maps` shows an 11,536 MB PROT_NONE G1 reservation (JRE-only image, no jcmd). - Recognition-result storms (~20 MB `owner_face_recognition` POSTs with 512-dim boxed embeddings + full-pool embeddings GET) push G1 to ~4 GiB committed, which it never uncommits (`G1PeriodicGC` default off) → RSS ratchets across container lifetime: 408 MB @5min fresh → 1.15 GiB @35min baseline → 4.68 GiB after 3 recognition transients in one evening. - Code audit found **zero unbounded retention structures** (SSE broadcaster self-cleans, no Caffeine/Guava caches; multipart spools to disk). - **Tail risk**: unbounded heap means a future loop regression races the host OOM-killer (JVM = fattest RSS). ## Fix One env line in `deploy/compose.yaml`: `JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=6 -XX:InitialRAMPercentage=2` (≈2.8 GiB max, overridable via `MEMORIES_API_JAVA_TOOL_OPTIONS`). ## Out of scope (deliberately) - `G1PeriodicGCInterval` uncommit tuning (aesthetic; rec #1 bounds RSS already) - Streaming the 20 MB result POST / paging the embeddings GET — YAGNI at 2–4 runs/day; revisit at ~10× pool size - Actuator `jvm_memory_used` metrics exposure — separate small change if wanted
chore(api): cap JVM heap at 6% of host via JAVA_TOOL_OPTIONS
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 2m33s
fb7f947bb6
Without an explicit -Xmx, the container default is MaxRAMPercentage=25 of
the HOST (api service has no cgroup memory limit) ≈ 11.7 GiB on the 46 GiB
deploy host. Recognition-result storms (~20 MB owner_face_recognition POSTs
with boxed embeddings, plus the full-pool embeddings GET) push G1 to ~4 GiB
committed which it never uncommits (G1PeriodicGC 0 = off), so RSS ratchets
across a container lifetime: measured 408 MB @5min fresh → 1.15 GiB @35min
baseline → 4.68 GiB after absorbing three recognition transients (oracle
diagnosis 2026-10-15: not a leak, zero unbounded retention structures in
code — SSE broadcaster self-cleans, no Caffeine/Guava caches).

Worse tail risk: with an unbounded heap a future loop regression would race
the host OOM-killer (JVM = fattest RSS). 6% ≈ 2.8 GiB max — generous for
this workload, deterministic RSS plateau. Overridable via
MEMORIES_API_JAVA_TOOL_OPTIONS for other hosts.
chris merged commit 01cde1dad6 into main 2026-10-15 22:41:50 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1246
No description provided.