perf: recognition-run pipeline latency + carousel gate rework #1040

Merged
chris merged 1 commit from fix/recognition-run-ux into main 2026-10-05 21:39:56 +02:00
Owner

What

One PR, five owner-approved items:

Persistence latency (measured on the production library, 21,309 faces / run):

  1. V154 migration - GIN index on class_instance_occurrence_candidates.occurrence_refs (jsonb_path_ops). 2.5 s build, 14 MB.
  2. Supersede pass - jsonb containment OR-chain replaces the 250-clause CAST(... AS VARCHAR) LIKE chain. Measured 88 s -> ~0.5 ms per chunk (86 chunks/run, ~80% of persistence time). H2 regression suite branches to the old LIKE chain; semantics identical.
  3. Rejected-pair check - SQL pushdown of the rejectedTargetedPair signature match: ~220 ms per B-card -> single indexed query (was ~80 s/run across 381 B-cards). H2 branches to the former in-memory scan.
  4. Worker timeout - result-submission read timeout 120 s -> 600 s. Every full-run submission (~3.8k candidates) exceeded 120 s and died on httpx.ReadTimeout, then rode the recovery retry, doubling wall time.

Carousel gate rework (owner decision 2026-10-05):

  1. Spinner only after reset - new GET /api/profile/people/recluster-gate reports resetRun (SQL: reset audit event newer than the newest terminal recognition job). Only reset runs hide the board; routine runs keep cards interactive behind a quiet background-refresh banner. 30 s background poll refreshes page 1 (new quality-score order) automatically when a run lands.

Tests

  • :apps:api:test full suite green (1618 tests, incl. Testcontainers Postgres migration chain through V154)
  • New: 3 Postgres integration tests for the resetRun derivation; 3 controller tests for the gate endpoint; store spec rewritten for the two-signal semantics (4 tests)
  • :apps:api:check green; svelte-check 0 errors; vitest 66/66; worker suite 249/249

Deploy scopes

api + workers + web (V154 migration runs on API boot)

## What One PR, five owner-approved items: **Persistence latency (measured on the production library, 21,309 faces / run):** 1. **V154 migration** - GIN index on class_instance_occurrence_candidates.occurrence_refs (jsonb_path_ops). 2.5 s build, 14 MB. 2. **Supersede pass** - jsonb containment OR-chain replaces the 250-clause CAST(... AS VARCHAR) LIKE chain. Measured 88 s -> ~0.5 ms per chunk (86 chunks/run, ~80% of persistence time). H2 regression suite branches to the old LIKE chain; semantics identical. 3. **Rejected-pair check** - SQL pushdown of the rejectedTargetedPair signature match: ~220 ms per B-card -> single indexed query (was ~80 s/run across 381 B-cards). H2 branches to the former in-memory scan. 4. **Worker timeout** - result-submission read timeout 120 s -> 600 s. Every full-run submission (~3.8k candidates) exceeded 120 s and died on httpx.ReadTimeout, then rode the recovery retry, doubling wall time. **Carousel gate rework (owner decision 2026-10-05):** 5. **Spinner only after reset** - new GET /api/profile/people/recluster-gate reports resetRun (SQL: reset audit event newer than the newest terminal recognition job). Only reset runs hide the board; routine runs keep cards interactive behind a quiet background-refresh banner. 30 s background poll refreshes page 1 (new quality-score order) automatically when a run lands. ## Tests - :apps:api:test full suite green (1618 tests, incl. Testcontainers Postgres migration chain through V154) - New: 3 Postgres integration tests for the resetRun derivation; 3 controller tests for the gate endpoint; store spec rewritten for the two-signal semantics (4 tests) - :apps:api:check green; svelte-check 0 errors; vitest 66/66; worker suite 249/249 ## Deploy scopes api + workers + web (V154 migration runs on API boot)
perf(api,workers,web): recognition-run pipeline latency + carousel gate rework
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 1m4s
fe27be0e62
Persistence latency (owner-reported 2-8 min per run, measured):
- V154: GIN index on class_instance_occurrence_candidates.occurrence_refs
  (jsonb_path_ops) — 2.5 s build, 14 MB on the production library.
- Supersede pass: per-face jsonb containment (@>) replaces the 250-clause
  CAST(... AS VARCHAR) LIKE chain — ~88 s per 250-face chunk -> ~0.5 ms
  (86 chunks per 21.3k-face run; the dominant cost). H2 regression suite
  (lacks @>) branches to the former portable LIKE chain; semantics
  identical.
- hasRejectedTargetedPair: SQL pushdown of the rejectedTargetedPair
  signature match (~220 ms per B-card full scan -> single indexed COUNT).
  H2 branches to the former in-memory scan.
- Worker result submission read timeout 120 s -> 600 s: every full-run
  submission (~3.8k candidates) exceeded 120 s and died on
  httpx.ReadTimeout, then rode the recovery retry — doubling the
  owner-visible run time.

Carousel gate rework (owner decision 2026-10-05):
- GET /api/profile/people/recluster-gate reports resetRun: true only while
  a run enqueued AFTER the last people_clustering_reset audit event has
  not been superseded by a terminal run (single SQL derivation).
- Spinner (board hidden) only for reset runs; routine runs keep cards
  interactive behind a quiet background-refresh banner.
- suggestionsStale store signal + 30 s background poll: page 1 refetches
  in the new quality-score order automatically when a run lands.
- Store/controller unit tests + Postgres integration tests for the
  resetRun derivation (reset->run->terminal transitions).

Version bump 1.21.65.
chris merged commit 89d0614da5 into main 2026-10-05 21:39:56 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1040
No description provided.