perf(api): fast timeline days list (kill correlated lateral + indexable layout range) #1009
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/timeline-days-query-speed"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Goal
GET /api/timeline/dayswarms in <150 ms (days-only) and the layout manifest query stays ≤100 ms. No API/semantic change.Root cause
LEFT JOIN LATERALoverupload_session_files(22.605 bitmap-heap probes per request) and a per-row OR-of-subplans visibility (owner_id IN (...) OR EXISTS (contributions ...)). PostgreSQL charges correlated subplans per row, pushing plan cost to 686.783 (scope=all) / 48.444.346 (scope=group) — far abovejit_above_cost=100000— so 440–550 ms of JIT compilation was paid per request before reading a single row. Baseline days = ~550 ms JIT + ~200 ms work (SET jit=offbaseline: 199 ms).Changes
NOT EXISTSwith a constant unsupported-format list; the planner builds one small hash (2.772 rows) once per request instead of probing per item. Option (b) — LATERAL +INCLUDE (detected_format)index — measured 283–357 ms vs 70–120 ms for the anti-join: 22.605 correlated index probes remain the bottleneck even as index-only scans. Invariant (live DB, 0 divergences over 28.154 items): everyupload_session_filesrow of onecommitted_media_item_idcarries the samedetected_format; the multi-row case is pinned by a regression test.LEFT JOIN media_owner_profiles viewer+LEFT JOIN (SELECT DISTINCT c.media_item_id ...) shared. Total plan cost ≈ 3.000–6.034 → no JIT. Thesharedsubquery is DISTINCT (one item can be in several of the viewer's groups); plain-join version multiplied day counts (28.968 vs 27.325 joined rows → 276 day rows with wrong counts; fixed, then 0 diffs both directions).day_dateonly,day_keycomputed once in projection/ORDER BY (DAY_KEY_FROM_DATE) — skips 22.605to_charcalls.captured:YYYY-MM-DD, a half-open UTC range (>= :layoutCaptureFrom AND < :layoutCaptureTo) is added ahead of the exactIN (:dayKeys)correctness filter → range scan instead of evaluating the computed day_key expression over all 27k active items.upload_session_files (committed_media_item_id) WHERE detected_format IS NOT NULL AND lower(...) NOT IN (...)): anti-join build side 57–59 ms → 0.8 ms. Measured locally insideBEGIN ... ROLLBACK; never applied to the live DB.layoutDayKeysparam (not needed).Live-DB evidence (warm medians, interleaved A/B, 5 runs; ~27.3k active items)
Candidate days plan: cost 6.022 (vs 686.783 baseline), planning 1.6 ms, no JIT section. Execution 97 ms on this run; fully warm medians above. After V152 the build side is 0.5–0.7 ms (index scan) instead of a 25–30 ms seq scan over 59.756 rows.
Tests
:apps:api:testgreen (all suites, 0 failures).upload_session_filesregression (repeated jpeg → included; repeated nef → excluded; mixed jpeg+nef → excluded, documenting the anti-join's "any unsupported row disqualifies" semantics), scopeall/group/myday-count regression through the join-backed visibility (own OR shared; revoked contribution and unrelated media never visible), updated SQL-shape assertions.Follow-up (ISSUE-094)
With a favorite filter active, the plan keeps a row-wise metadata_overrides predicate whose cost can re-cross
jit_above_cost→ filtered requests slow again. Unfiltered path is fixed; filtered path is tracked in KNOWN_ISSUES.