fix(ci): serialize memory-heavy test jobs + cap test JVM heap — kill the recurring testcontainers-ryuk OOM (resolves ISSUE-062) #836

Merged
chris merged 1 commit from fix/ci-oom-serialization into main 2026-09-23 21:35:52 +02:00
Owner

Problem

4th occurrence of OutOfMemoryError … thread testcontainers-ryuk killing scoped PR tests (runs 2707, 2740, 2749 + v1.21.3's build OOM). Root cause: the host runs TWO memories runners (each capacity 1, same label) on one 15 GiB host — memory-heavy test jobs overlap and exhaust RAM.

Fix

  1. Host-file flock serialization (chosen over job-level concurrency: — schema-verified that Forgejo v16 stores concurrency per RUN only, so job-level would be silently ignored; a workflow-level group would also serialize deploys): new scripts/ci/acquire-heavy-test-lock.sh takes an exclusive fd flock on the shared host mount, sourced by all 16 heavy steps in build-deploy.yml + 4 in daily-full-tests.yml. Container death auto-releases (kernel closes fd) → no stale locks; bounded 60-min polling with clear failure instead of hanging into the 3h runner timeout.
  2. Bounded test JVMs (root shared Gradle config, covers apps:api + worker-media): maxHeapSize 2g, MaxDirectMemorySize 1g, maxParallelForks 1 — verified on the JVM command line via --info.
  3. ISSUE-062 archived as resolved with fix reference.

Verification

YAML valid (both workflows); flock mutual exclusion proven on the host AND inside the real job image (docker:28-cli with the same /home/chris/git mount); Gradle config compiles + Pca2DTest green. Honest caveat: the OOM itself can only be disproven by concurrent CI runs over time — this is the known-good mitigation pattern.

Merge note: PRs whose branches predate this fix (#835) need main merged in to pick up the lock before their CI rerun.

## Problem 4th occurrence of `OutOfMemoryError … thread testcontainers-ryuk` killing scoped PR tests (runs 2707, 2740, 2749 + v1.21.3's build OOM). Root cause: the host runs TWO memories runners (each capacity 1, same label) on one 15 GiB host — memory-heavy test jobs overlap and exhaust RAM. ## Fix 1. **Host-file flock serialization** (chosen over job-level `concurrency:` — schema-verified that Forgejo v16 stores concurrency per RUN only, so job-level would be silently ignored; a workflow-level group would also serialize deploys): new `scripts/ci/acquire-heavy-test-lock.sh` takes an exclusive fd flock on the shared host mount, sourced by all 16 heavy steps in build-deploy.yml + 4 in daily-full-tests.yml. Container death auto-releases (kernel closes fd) → no stale locks; bounded 60-min polling with clear failure instead of hanging into the 3h runner timeout. 2. **Bounded test JVMs** (root shared Gradle config, covers apps:api + worker-media): `maxHeapSize 2g`, `MaxDirectMemorySize 1g`, `maxParallelForks 1` — verified on the JVM command line via --info. 3. ISSUE-062 archived as resolved with fix reference. ## Verification YAML valid (both workflows); flock mutual exclusion proven on the host AND inside the real job image (docker:28-cli with the same /home/chris/git mount); Gradle config compiles + Pca2DTest green. Honest caveat: the OOM itself can only be disproven by concurrent CI runs over time — this is the known-good mitigation pattern. Merge note: PRs whose branches predate this fix (#835) need main merged in to pick up the lock before their CI rerun.
fix(ci): serialize memory-heavy test jobs + cap test JVM heap — kill the recurring testcontainers-ryuk OOM (resolves ISSUE-062)
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 1m45s
46fd351877
chris merged commit 38c015a16d into main 2026-09-23 21:35:52 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!836
No description provided.