fix(ci): serialize memory-heavy test jobs + cap test JVM heap — kill the recurring testcontainers-ryuk OOM (resolves ISSUE-062) #836
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/ci-oom-serialization"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
4th occurrence of
OutOfMemoryError … thread testcontainers-ryukkilling scoped PR tests (runs 2707, 2740, 2749 + v1.21.3's build OOM). Root cause: the host runs TWO memories runners (each capacity 1, same label) on one 15 GiB host — memory-heavy test jobs overlap and exhaust RAM.Fix
concurrency:— schema-verified that Forgejo v16 stores concurrency per RUN only, so job-level would be silently ignored; a workflow-level group would also serialize deploys): newscripts/ci/acquire-heavy-test-lock.shtakes an exclusive fd flock on the shared host mount, sourced by all 16 heavy steps in build-deploy.yml + 4 in daily-full-tests.yml. Container death auto-releases (kernel closes fd) → no stale locks; bounded 60-min polling with clear failure instead of hanging into the 3h runner timeout.maxHeapSize 2g,MaxDirectMemorySize 1g,maxParallelForks 1— verified on the JVM command line via --info.Verification
YAML valid (both workflows); flock mutual exclusion proven on the host AND inside the real job image (docker:28-cli with the same /home/chris/git mount); Gradle config compiles + Pca2DTest green. Honest caveat: the OOM itself can only be disproven by concurrent CI runs over time — this is the known-good mitigation pattern.
Merge note: PRs whose branches predate this fix (#835) need main merged in to pick up the lock before their CI rerun.