perf(api): set-based candidate overlap + batched confirm — remove per-decision N+1 (890 face queries → 2) #821

Merged
chris merged 1 commit from fix/decision-perf-api into main 2026-09-22 21:58:57 +02:00
Owner

Problem

Every tentative accept/reject ran extraOverlapCandidateIds with one SQL query per face-bearing pending candidate (measured: 890 queries on the main owner) plus a 26 MB payload JSONB over-fetch; confirmTentativeCandidates repeated the full overlap resolution per selected candidate (select-all on a 649-candidate person ≈ 700k statements). Hot reads were already indexed — this was algorithmic, not missing indexes.

Changes

  • pendingCandidateRows: only needed columns (no payload), excludes face_detection artifacts (media-only refs can never overlap review candidates).
  • Set-based overlap: all face→media lookups prefetched in one chunked query (1000 ids/chunk); overlap computed in memory; null-safe media comparison identical to the old row comparison.
  • confirmTentativeCandidates: single batch pass resolves all per-candidate lock sets once (chunked ref-intersection + candidate read); accept primitive keeps per-row locking/revalidation and per-candidate success semantics.
  • V134 partial composite index on class_instance_occurrences (WHERE state='accepted', predicate chosen so both runtime assignment queries can use it).
  • New batch extraOverlapCandidateIds(List<CandidateSnapshot>, …) default on DomainCandidateDecisionStrategy — only people overrides it; other domains untouched.

Tests

  • New CountingJdbcClient test decorator; constant-statement scan test; one-overlap-resolution confirm test; ref-intersecting-only confirm; differential parity vs. an in-test legacy oracle on a 25-candidate corpus; V134 migration content tests (H2 + Postgres container).
  • dev.memories.api.discovery.*: 464 tests, 0 failures. *Decision*/*Candidate*: 156 tests, 0 failures.

Expected effect

Decision POSTs drop from ~890 queries + 26 MB detoast to ~2 batched statements + small refs; multi-select confirm no longer multiplies. Complements #820 (frontend optimistic decisions). Follow-up (separate): R4 recognition re-arm debounce/incrementalization.

## Problem Every tentative accept/reject ran `extraOverlapCandidateIds` with one SQL query per face-bearing pending candidate (measured: **890 queries** on the main owner) plus a **26 MB** `payload` JSONB over-fetch; `confirmTentativeCandidates` repeated the full overlap resolution **per selected candidate** (select-all on a 649-candidate person ≈ 700k statements). Hot reads were already indexed — this was algorithmic, not missing indexes. ## Changes - `pendingCandidateRows`: only needed columns (no payload), excludes `face_detection` artifacts (media-only refs can never overlap review candidates). - Set-based overlap: all face→media lookups prefetched in one chunked query (1000 ids/chunk); overlap computed in memory; null-safe media comparison identical to the old row comparison. - `confirmTentativeCandidates`: single batch pass resolves all per-candidate lock sets once (chunked ref-intersection + candidate read); accept primitive keeps per-row locking/revalidation and per-candidate success semantics. - V134 partial composite index on `class_instance_occurrences` (`WHERE state='accepted'`, predicate chosen so both runtime assignment queries can use it). - New batch `extraOverlapCandidateIds(List<CandidateSnapshot>, …)` default on `DomainCandidateDecisionStrategy` — only people overrides it; other domains untouched. ## Tests - New `CountingJdbcClient` test decorator; constant-statement scan test; one-overlap-resolution confirm test; ref-intersecting-only confirm; differential parity vs. an in-test legacy oracle on a 25-candidate corpus; V134 migration content tests (H2 + Postgres container). - `dev.memories.api.discovery.*`: 464 tests, 0 failures. `*Decision*`/`*Candidate*`: 156 tests, 0 failures. ## Expected effect Decision POSTs drop from ~890 queries + 26 MB detoast to ~2 batched statements + small refs; multi-select confirm no longer multiplies. Complements #820 (frontend optimistic decisions). Follow-up (separate): R4 recognition re-arm debounce/incrementalization.
perf(api): set-based candidate overlap + batched confirm — remove per-decision N+1 (890 face queries → 2)
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 1m47s
b5cd1bc492
chris merged commit 973e8c412e into main 2026-09-22 21:58:57 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!821
No description provided.