perf(api): fast timeline days list (kill correlated lateral + indexable layout range) #1009

Merged
chris merged 1 commit from feat/timeline-days-query-speed into main 2026-10-03 19:57:49 +02:00
Owner

Goal

GET /api/timeline/days warms in <150 ms (days-only) and the layout manifest query stays ≤100 ms. No API/semantic change.

Root cause

  • The days query carried a correlated LEFT JOIN LATERAL over upload_session_files (22.605 bitmap-heap probes per request) and a per-row OR-of-subplans visibility (owner_id IN (...) OR EXISTS (contributions ...)). PostgreSQL charges correlated subplans per row, pushing plan cost to 686.783 (scope=all) / 48.444.346 (scope=group) — far above jit_above_cost=100000 — so 440–550 ms of JIT compilation was paid per request before reading a single row. Baseline days = ~550 ms JIT + ~200 ms work (SET jit=off baseline: 199 ms).

Changes

  1. Format gate → hash anti-join (option a, measured over option b). The LATERAL is replaced by NOT EXISTS with a constant unsupported-format list; the planner builds one small hash (2.772 rows) once per request instead of probing per item. Option (b) — LATERAL + INCLUDE (detected_format) index — measured 283–357 ms vs 70–120 ms for the anti-join: 22.605 correlated index probes remain the bottleneck even as index-only scans. Invariant (live DB, 0 divergences over 28.154 items): every upload_session_files row of one committed_media_item_id carries the same detected_format; the multi-row case is pinned by a regression test.
  2. Visibility → join-backed. LEFT JOIN media_owner_profiles viewer + LEFT JOIN (SELECT DISTINCT c.media_item_id ...) shared. Total plan cost ≈ 3.000–6.034 → no JIT. The shared subquery is DISTINCT (one item can be in several of the viewer's groups); plain-join version multiplied day counts (28.968 vs 27.325 joined rows → 276 day rows with wrong counts; fixed, then 0 diffs both directions).
  3. Days aggregation groups by day_date only, day_key computed once in projection/ORDER BY (DAY_KEY_FROM_DATE) — skips 22.605 to_char calls.
  4. Layout manifest: capture range. When every window key matches captured:YYYY-MM-DD, a half-open UTC range (>= :layoutCaptureFrom AND < :layoutCaptureTo) is added ahead of the exact IN (:dayKeys) correctness filter → range scan instead of evaluating the computed day_key expression over all 27k active items.
  5. V152 migration (partial index on upload_session_files (committed_media_item_id) WHERE detected_format IS NOT NULL AND lower(...) NOT IN (...)): anti-join build side 57–59 ms → 0.8 ms. Measured locally inside BEGIN ... ROLLBACK; never applied to the live DB.
  6. Skipped: separate contributions partial index (contribution join is one ~12–16 ms hash build, no >50 ms gain); optional phase-2 layoutDayKeys param (not needed).

Live-DB evidence (warm medians, interleaved A/B, 5 runs; ~27.3k active items)

Query Before (median) After (median)
(a) days-only 704 ms 76 ms
(b) layout manifest 523 ms 22 ms
(c) sum ~1.227 s ~98 ms

Candidate days plan: cost 6.022 (vs 686.783 baseline), planning 1.6 ms, no JIT section. Execution 97 ms on this run; fully warm medians above. After V152 the build side is 0.5–0.7 ms (index scan) instead of a 25–30 ms seq scan over 59.756 rows.

Tests

  • :apps:api:test green (all suites, 0 failures).
  • New: multi-row upload_session_files regression (repeated jpeg → included; repeated nef → excluded; mixed jpeg+nef → excluded, documenting the anti-join's "any unsupported row disqualifies" semantics), scope all/group/my day-count regression through the join-backed visibility (own OR shared; revoked contribution and unrelated media never visible), updated SQL-shape assertions.

Follow-up (ISSUE-094)

With a favorite filter active, the plan keeps a row-wise metadata_overrides predicate whose cost can re-cross jit_above_cost → filtered requests slow again. Unfiltered path is fixed; filtered path is tracked in KNOWN_ISSUES.

## Goal `GET /api/timeline/days` warms in <150 ms (days-only) and the layout manifest query stays ≤100 ms. No API/semantic change. ## Root cause - The days query carried a **correlated `LEFT JOIN LATERAL` over `upload_session_files`** (22.605 bitmap-heap probes per request) *and* a per-row **OR-of-subplans visibility** (`owner_id IN (...) OR EXISTS (contributions ...)`). PostgreSQL charges correlated subplans per row, pushing plan cost to 686.783 (scope=all) / 48.444.346 (scope=group) — far above `jit_above_cost=100000` — so **440–550 ms of JIT compilation** was paid per request before reading a single row. Baseline days = ~550 ms JIT + ~200 ms work (`SET jit=off` baseline: 199 ms). ## Changes 1. **Format gate → hash anti-join (option a, measured over option b).** The LATERAL is replaced by `NOT EXISTS` with a constant unsupported-format list; the planner builds one small hash (2.772 rows) once per request instead of probing per item. Option (b) — LATERAL + `INCLUDE (detected_format)` index — measured 283–357 ms vs 70–120 ms for the anti-join: 22.605 correlated index probes remain the bottleneck even as index-only scans. Invariant (live DB, 0 divergences over 28.154 items): every `upload_session_files` row of one `committed_media_item_id` carries the same `detected_format`; the multi-row case is pinned by a regression test. 2. **Visibility → join-backed.** `LEFT JOIN media_owner_profiles viewer` + `LEFT JOIN (SELECT DISTINCT c.media_item_id ...) shared`. Total plan cost ≈ 3.000–6.034 → no JIT. The `shared` subquery is **DISTINCT** (one item can be in several of the viewer's groups); plain-join version multiplied day counts (28.968 vs 27.325 joined rows → 276 day rows with wrong counts; fixed, then 0 diffs both directions). 3. **Days aggregation groups by `day_date` only**, `day_key` computed once in projection/ORDER BY (`DAY_KEY_FROM_DATE`) — skips 22.605 `to_char` calls. 4. **Layout manifest: capture range.** When every window key matches `captured:YYYY-MM-DD`, a half-open UTC range (`>= :layoutCaptureFrom AND < :layoutCaptureTo`) is added ahead of the exact `IN (:dayKeys)` correctness filter → range scan instead of evaluating the computed day_key expression over all 27k active items. 5. **V152 migration** (partial index on `upload_session_files (committed_media_item_id) WHERE detected_format IS NOT NULL AND lower(...) NOT IN (...)`): anti-join build side 57–59 ms → 0.8 ms. Measured locally inside `BEGIN ... ROLLBACK`; never applied to the live DB. 6. **Skipped:** separate contributions partial index (contribution join is one ~12–16 ms hash build, no >50 ms gain); optional phase-2 `layoutDayKeys` param (not needed). ## Live-DB evidence (warm medians, interleaved A/B, 5 runs; ~27.3k active items) | Query | Before (median) | After (median) | |---|---|---| | (a) days-only | 704 ms | **76 ms** | | (b) layout manifest | 523 ms | **22 ms** | | (c) sum | ~1.227 s | **~98 ms** | Candidate days plan: cost 6.022 (vs 686.783 baseline), planning 1.6 ms, no JIT section. Execution 97 ms on this run; fully warm medians above. After V152 the build side is 0.5–0.7 ms (index scan) instead of a 25–30 ms seq scan over 59.756 rows. ## Tests - `:apps:api:test` green (all suites, 0 failures). - New: multi-row `upload_session_files` regression (repeated jpeg → included; repeated nef → excluded; mixed jpeg+nef → excluded, documenting the anti-join's "any unsupported row disqualifies" semantics), scope `all`/`group`/`my` day-count regression through the join-backed visibility (own OR shared; revoked contribution and unrelated media never visible), updated SQL-shape assertions. ## Follow-up (ISSUE-094) With a **favorite** filter active, the plan keeps a row-wise metadata_overrides predicate whose cost can re-cross `jit_above_cost` → filtered requests slow again. Unfiltered path is fixed; filtered path is tracked in KNOWN_ISSUES.
perf(api): fast timeline days list (kill correlated lateral + indexable layout range)
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 3m11s
11760a861b
Days list: replace the correlated LEFT JOIN LATERAL over upload_session_files
(22.605 bitmap-heap probes/request, and its per-row cost charge pushed plan
cost past jit_above_cost=100000 -> 440-550 ms JIT compile per request) with a
hash anti-join over a constant unsupported-format list. Replace the per-row
OR visibility (owner_id IN (...) OR EXISTS (contributions ...)) with join-backed
visibility (LEFT JOIN media_owner_profiles viewer + LEFT JOIN SELECT DISTINCT
contributions/shared) so plan cost falls from 686.783 (all) / 48.444.346
(group) to ~6k and JIT never triggers. Group day counts by day_date and compute
day_key once in the projection/ORDER BY (skips 22.605 to_char calls).

Layout manifest: narrow with a half-open UTC capture range derived from the
window's day keys (only when every key is captured:YYYY-MM-DD) ahead of the
exact IN (:dayKeys) correctness filter, so the planner range-scans
capture_timestamp instead of evaluating the day_key expression over all 27k
active items.

Invariant (measured 2026-10-03, live DB, 0 divergences over 28.154 items):
every upload_session_files row of one committed_media_item_id carries the same
detected_format, so the anti-join and the old LIMIT 1 lateral pick identical
items; the multi-row case is pinned by new tests. The DISTINCT shared join
prevents double-counting items contributed to several of the viewer's groups.

V152 adds a partial index on upload_session_files (committed_media_item_id)
restricted to unsupported rows (2.772 of 59.756): anti-join build side
57-59 ms -> 0.8 ms.

Live warm medians (interleaved A/B, 5 runs): days 704 ms -> 76 ms; layout
523 ms -> 22 ms; two-request total -> under 100 ms (both targets met).

Tests: multi-row upload_session_files regression (repeated supported/
unsupported rows; mixed-format divergence documented), scope all/group/my
visibility regression through the new join-backed predicate, updated SQL-shape
assertions.
chris merged commit d4866a8834 into main 2026-10-03 19:57:49 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1009
No description provided.