fix(api): bound worker_jobs history with 30d purge + index-covered ProcessingSummary #1251

Merged
chris merged 2 commits from fix/worker-jobs-retention-summary-cost into main 2026-10-16 00:11:18 +02:00
Owner

Problem

Production (2026-10-15): worker_jobs grew unbounded to 992,096 rows / 6.7 GB of lifetime churn (plant_detection 255k+115k, discovery_people 79k+5k, thumbnail jobs in the hundreds of thousands, owner_face_recognition 45k+11k). The per-owner ProcessingSummary lane aggregates (COALESCE(SUM(CASE WHEN status …)), joined via mi.id = wj.media_item_id) scan that history on every web poll / SSE refetch → sustained ~109 % CPU on the postgres container with DataFileRead IO waits.

Fix (2 parts)

1. Retention — WorkerJobHistoryPurgeService: batched purge of terminal jobs older than 30 days (@Scheduled, startup+60 s, then every 10 min; 2,000 rows/batch, each batch its own REQUIRES_NEW transaction, capped at 50 batches/tick → initial 960k backlog drains over ~10 ticks without long transactions).

Safety rules (load-bearing, each pinned by tests):

  • failed/expired are NEVER purged — WorkerJobRecoveryService retries them (attempts < 3) and the enqueue anti-joins in WorkerJobEnqueueService use old failed rows as permanent-failure guards; purging them would convert guards into infinite enqueue loops.
  • Children deleted explicitly first (no ON DELETE CASCADE): worker_job_results, worker_artifacts, worker_media_access_grants (NOT NULL FKs) + processing_issues pointer rows (no FK → would orphan).
  • 30 days covers every history consumer: ProcessingSummary batch window (7 d), JobRunsService outcome window (7 d), result resubmit dedupe (non-terminal only).
  • No tombstone table (reset flows falsify input_version tombstones); the bounded table IS the retention policy.

2. Aggregate cost — V166 indexes (CREATE INDEX only, no rewrite, no migration-time DELETE):

  • idx_worker_jobs_media_type_status_updated (media_item_id, job_type, status, updated_at) → makes the lane aggregates index-only via the media join. Explicitly NOT owner-first: worker_jobs.owner_id is NULL for all media-scoped enqueues (V121 backfilled non-terminal only) — an owner-first index would serve empty ranges = silently wrong counts.
  • Partial index for the purge batch selection (predicate mirrors PURGEABLE_STATUSES, pinned by a test reading pg_get_indexdef).
  • Child-table FK lookup indexes for the batched deletes.

Rejected: Caffeine cache (new dependency, staleness vs SSE refetch), rollup table (transition-sync failure modes), partitioning (overkill while purge bounds the table — revisit only if steady-state still hurts).

Verification

  • WorkerJobHistoryPurgeServicePostgresTest (Testcontainers PG, full Flyway chain V001→V166 on the real deploy/postgres image): 9 tests — strict < retention boundary, queued/leased/running/failed/expired untouched, both cancelled/canceled spellings, child cleanup + surviving-job children kept, idempotency, batch limit, index/service status-list sync.
  • EXPLAIN (ANALYZE, BUFFERS) on 500k-row synthetic prod-like churn: lane aggregate 56 ms → 23 ms after purge (2.4×); Index-Only-Scan path confirmed.
  • rtk ./gradlew :apps:api:test: 2094 tests, 0 failures.

Out of scope

Partitioning, archiving purged rows, the 12 semantic_embedding ml_image_unreadable failures (corrupt input images, terminal after 3 attempts).

## Problem Production (2026-10-15): `worker_jobs` grew unbounded to **992,096 rows / 6.7 GB** of lifetime churn (plant_detection 255k+115k, discovery_people 79k+5k, thumbnail jobs in the hundreds of thousands, owner_face_recognition 45k+11k). The per-owner ProcessingSummary lane aggregates (`COALESCE(SUM(CASE WHEN status …))`, joined via `mi.id = wj.media_item_id`) scan that history on **every web poll / SSE refetch** → sustained **~109 % CPU** on the postgres container with DataFileRead IO waits. ## Fix (2 parts) **1. Retention — `WorkerJobHistoryPurgeService`**: batched purge of terminal jobs older than 30 days (@Scheduled, startup+60 s, then every 10 min; 2,000 rows/batch, each batch its own REQUIRES_NEW transaction, capped at 50 batches/tick → initial 960k backlog drains over ~10 ticks without long transactions). Safety rules (load-bearing, each pinned by tests): - **`failed`/`expired` are NEVER purged** — WorkerJobRecoveryService retries them (`attempts < 3`) and the enqueue anti-joins in `WorkerJobEnqueueService` use old failed rows as permanent-failure guards; purging them would convert guards into infinite enqueue loops. - Children deleted explicitly first (no ON DELETE CASCADE): `worker_job_results`, `worker_artifacts`, `worker_media_access_grants` (NOT NULL FKs) + `processing_issues` pointer rows (no FK → would orphan). - 30 days covers every history consumer: ProcessingSummary batch window (7 d), JobRunsService outcome window (7 d), result resubmit dedupe (non-terminal only). - No tombstone table (reset flows falsify `input_version` tombstones); the bounded table IS the retention policy. **2. Aggregate cost — V166 indexes** (CREATE INDEX only, no rewrite, no migration-time DELETE): - `idx_worker_jobs_media_type_status_updated (media_item_id, job_type, status, updated_at)` → makes the lane aggregates index-only via the media join. **Explicitly NOT owner-first**: `worker_jobs.owner_id` is NULL for all media-scoped enqueues (V121 backfilled non-terminal only) — an owner-first index would serve *empty ranges* = silently wrong counts. - Partial index for the purge batch selection (predicate mirrors `PURGEABLE_STATUSES`, pinned by a test reading `pg_get_indexdef`). - Child-table FK lookup indexes for the batched deletes. Rejected: Caffeine cache (new dependency, staleness vs SSE refetch), rollup table (transition-sync failure modes), partitioning (overkill while purge bounds the table — revisit only if steady-state still hurts). ## Verification - `WorkerJobHistoryPurgeServicePostgresTest` (Testcontainers PG, full Flyway chain V001→V166 on the real deploy/postgres image): 9 tests — strict `<` retention boundary, queued/leased/running/failed/expired untouched, both `cancelled`/`canceled` spellings, child cleanup + surviving-job children kept, idempotency, batch limit, index/service status-list sync. - EXPLAIN (ANALYZE, BUFFERS) on 500k-row synthetic prod-like churn: lane aggregate 56 ms → **23 ms after purge (2.4×)**; Index-Only-Scan path confirmed. - `rtk ./gradlew :apps:api:test`: **2094 tests, 0 failures**. ## Out of scope Partitioning, archiving purged rows, the 12 `semantic_embedding ml_image_unreadable` failures (corrupt input images, terminal after 3 attempts).
Production evidence 2026-10-15: worker_jobs at 992,096 rows / 6.7 GB of
lifetime churn; postgres at ~109% CPU with the per-owner ProcessingSummary
lane aggregate (COALESCE(SUM(CASE ...)) per lane, joined via media_items)
hammered by web polling + SSE refetches.

Two-part fix, no UI-visible behavior change:

1. V166 indexes (CREATE INDEX only, no table rewrite):
   - idx_worker_jobs_media_type_status_updated (media_item_id, job_type,
     status, updated_at): media-first key matches the mi.id = wj.media_item_id
     join so each lane aggregate reads contiguous per-media index ranges.
     Deliberately NOT owner-scoped: V121 backfilled owner_id only for
     non-terminal rows and media-scoped enqueues leave it NULL — an
     owner-first index would serve empty ranges (silently wrong counts).
   - idx_worker_jobs_terminal_updated_at: partial index over exactly the
     purgeable statuses (kept in sync with the service list, pinned by test).
   - child-table indexes for the FK-enforced batch deletes
     (worker_job_results / worker_artifacts / worker_media_access_grants)
     plus a partial index for processing_issues worker_job pointer rows.

2. WorkerJobHistoryPurgeService (@Scheduled 10 min, startup +60s):
   deletes terminal worker_jobs (completed, completed_with_issues,
   cancelled/canceled) older than 30 days in REQUIRES_NEW batches of 2,000
   (children first: worker_job_results, worker_artifacts,
   worker_media_access_grants, processing_issues pointer rows; worker_jobs
   has no ON DELETE CASCADE children). NEVER touches queued/leased/running
   (live queue) or failed/expired (recovery retry state + deliberate
   permanent-failure guards in enqueue anti-joins — purging those would
   create infinite re-enqueue loops). No tombstone table: reset flows
   re-enqueue with fresh input_versions, which falsifies any
   (media, type, input_version) tombstone — the bounded table IS the
   retention policy. 30d comfortably covers the 7d recent-batch window
   (ProcessingSummary), the 7d JobRunsService outcome window, and the
   WorkerResultService resubmit dedupe. Nullable provenance references
   (discovery_results.worker_job_id,
   class_instance_occurrence_candidates.worker_job_id) are write-side-only
   and left dangling by design — no read path joins them to worker_jobs.

Operator status: worker_job_history_purge idleScheduler row in
GET /api/admin/job-runs. Config: memories.worker.history-purge.retention-days
(default 30) and .batch-size (default 2000).

Tests: WorkerJobHistoryPurgeServicePostgresTest runs the full Flyway chain
on Testcontainers PG — pins retention boundary (strict <), terminal-status
safety (queued/leased/running/failed/expired untouched), both cancelled
spellings, child-row cleanup, surviving-job children kept, idempotency,
batch limit, and the partial-index/service status-list sync.
EXPLAIN (ANALYZE, BUFFERS) on a 500k-row synthetic worker_jobs: lane
aggregate 56 ms unbounded -> 23 ms purged (2.4x), Index Only Scan path
confirmed; aggregate cost is now proportional to the bounded window, not
lifetime history.

KNOWN_ISSUES: nothing left open — bounded retention + covering index close
the growth and CPU regressions; partitioning deliberately not pursued
(family-install complexity outweighs benefit while the table is bounded).
fix(api): cap worker_jobs history purge at 50 batches per tick
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 2m51s
576ce3fef7
Bounds the scheduled drain: the initial ~960k-row production backlog now
clears over consecutive 10-min ticks (50 x 2000 rows each, ~10 ticks) with
per-batch commits, instead of one unbounded while-loop per tick. Also caps
the worst case if the purgeable-status list were ever misconfigured.
chris merged commit 7b7b97247f into main 2026-10-16 00:11:18 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1251
No description provided.