feat(api): server-authoritative change feed (spec 065 PR-A) #1352

Merged
chris merged 1 commit from feat/sync-change-feed into main 2026-10-24 13:05:37 +02:00
Owner

Implements spec 065 increment PR-A — server change feed per ADR 0117 Option D. Additive and read-only: nothing consumes the feed yet (no Android, no intents, no existing-endpoint behavior change).

Scope (S1–S3 server side only)

  • V176 migration: media_items.change_seq BIGINT NOT NULL, one-time backfill in per-owner created_at order (updated_at only as deterministic tiebreaker — it is NOT the cursor going forward), single-row global sync_seq counter table, per-owner media_owner_profiles.sync_epoch (starts at 1), keyset index (owner_id, change_seq).
  • GET /api/sync/changes?cursor=&limit= (S2): owner-scoped keyset pagination (WHERE change_seq > cursor ORDER BY change_seq ASC LIMIT), envelope {epoch, changes[], nextCursor, hasMore}, per-change payload exactly the S2 fields (changeSeq, mediaItemId, lifecycleState, contentHash, originalFilename, byteSize, mediaType, capturedAt, updatedAt), state-feed semantics (the row's LATEST entry is the truth — trash/purge are lifecycle transitions on permanent rows, replay-from-0 always possible), first sync = cursor 0 (no bootstrap endpoint), default/max limit 200/1000, negative cursor → RFC 9457 ProblemDetail 400. Auth identical to sibling owner-scoped endpoints (/api/** authenticated + media:read API-credential scope, owner from session).
  • Epoch: clear-library bumps sync_epoch in the same transaction as the hard DELETE (the only signal that feed history was truncated). DB restore = operator adds 1 (ADR 0117).
  • Contract: path + SyncChangesPage/SyncChange schemas added to specs/001-v1-baseline/contracts/core-api.openapi.yaml following the file's existing style.

How commit-ordered seq allocation is guaranteed (ADR 0117 risk #1)

ONE coherent mechanism (documented in the migration header): a BEFORE INSERT OR UPDATE OF <sync-visible columns> trigger on media_items allocates change_seq from the single-row sync_seq counter via an atomic INSERT … ON CONFLICT DO UPDATE … RETURNING. The counter row lock is held to commit, so allocation order == commit order — the reason a plain Postgres SEQUENCE was rejected (non-transactional nextval() can commit out of order and burn gaps, which would make a client's > cursor keyset skip a committed change forever). The trigger deliberately covers INSERT too (an update-only trigger would never fire for new rows), so every writer goes through one code path. Explicit seq assignments (the backfill) pass through via an IS DISTINCT FROM OLD guard — a bare IS NOT NULL check would silently swallow every UPDATE because NEW.change_seq carries the row's previous value. The allocator self-heals an empty counter (DB restore) instead of failing with a not-null violation.

Writer inventory (verified exhaustively; every writer hits the trigger)

# Writer Statement File:line (main)
1 Upload commit INSERT INTO media_items apps/api/src/main/java/dev/memories/api/upload/UploadCommitService.java:280
2 Trash UPDATE lifecycle → trashed apps/api/src/main/java/dev/memories/api/media/TrashService.java:48
3 Restore from trash UPDATE lifecycle → active apps/api/src/main/java/dev/memories/api/media/TrashService.java:68
4 Permanent delete (single) UPDATE lifecycle → permanently_deleted apps/api/src/main/java/dev/memories/api/media/TrashService.java:92
5 Empty trash (bulk purge) UPDATE lifecycle → permanently_deleted (set-based) apps/api/src/main/java/dev/memories/api/media/TrashService.java:134
6 Worker metadata acceptance UPDATE sync-visible columns (capture/width/height/location/status) apps/api/src/main/java/dev/memories/api/worker/WorkerResultService.java:807 (acceptMetadataCandidate)
7 Clear library hard DELETE FROM media_items + epoch bump (same tx) apps/api/src/main/java/dev/memories/api/media/MediaMaintenanceService.java:1378 (delete) / :1397 (epoch)

Grep sweep over apps/api (all UPDATE|INSERT INTO|DELETE FROM media_items occurrences incl. multi-line text blocks) and apps/worker-media (no direct DB writes) found no other writer. Non-sync-visible UPDATEs (none exist today besides maintenance deletes) would not fire the trigger by design — the watched-column list matches the S2 payload; updated_at is watched, so any future service write that touches a sync field via updated_at is covered as a safety net.

Tests (SyncChangeFeedIntegrationTest — real Flyway chain, Testcontainers Postgres, TrashServiceTest-style harness)

  • (a) create→trash→restore→purge: strictly increasing seqs, correct lifecycle states, single latest-state entry on replay, physical row preserved.
  • (b) keyset pagination across limit boundaries (7 items / limit 3 → 3 pages, no dup/gap, sorted, epoch on every page, exhausted-cursor pull stable) + stability under a concurrent new change (late row lands strictly above the taken cursor).
  • (c) owner scoping: A never sees B's item or B's trash transition.
  • (d) writer-inventory test (the ADR-demanded one): exercises writers 1–6 above end-to-end through the real services and asserts a seq bump per writer; writer 7 (hard DELETE) is pinned by the epoch test since deleted rows cannot carry a seq.
  • (e) clear-library bumps owner epoch (+1), unrelated owner's epoch untouched, post-clear pull from 0 is empty.
  • (f) trash = UPDATE (same row, new seq); purge = lifecycle-state change, no physical row loss (COUNT(*)=1 after purge).
  • Rollback atomicity: a failed insert allocates no seq (counter unchanged). Payload fidelity (all S2 fields incl. contentHash from stored_objects). Negative cursor → 400.

Validation

  • rtk ./gradlew :apps:api:test: BUILD SUCCESSFUL — 2269 tests, 0 failures, 0 errors, 9 skipped (pre-existing conditional skips).
  • Note: a first run showed 2 failures in PersonRelationServicePostgresTest.theChosenRepresentativeWinsOverTheLatestOccurrence; reproduced identically on a pristine checkout of base e774b693 → pre-existing flake, already tracked as ISSUE-169; passes in isolation.

Not done (per increment plan)

No Android changes, no POST /api/sync/intents (PR-C), no event-log table, no per-device server state, no changes to existing endpoints. KNOWN_ISSUES.md untouched (no user-visible behavior ships).

Implements spec 065 increment **PR-A — server change feed** per ADR 0117 Option D. Additive and read-only: nothing consumes the feed yet (no Android, no intents, no existing-endpoint behavior change). ## Scope (S1–S3 server side only) - **V176 migration**: `media_items.change_seq BIGINT NOT NULL`, one-time backfill in per-owner `created_at` order (`updated_at` only as deterministic tiebreaker — it is NOT the cursor going forward), single-row **global** `sync_seq` counter table, per-owner `media_owner_profiles.sync_epoch` (starts at 1), keyset index `(owner_id, change_seq)`. - **`GET /api/sync/changes?cursor=&limit=`** (S2): owner-scoped keyset pagination (`WHERE change_seq > cursor ORDER BY change_seq ASC LIMIT`), envelope `{epoch, changes[], nextCursor, hasMore}`, per-change payload exactly the S2 fields (changeSeq, mediaItemId, lifecycleState, contentHash, originalFilename, byteSize, mediaType, capturedAt, updatedAt), state-feed semantics (the row's LATEST entry is the truth — trash/purge are lifecycle transitions on permanent rows, replay-from-0 always possible), first sync = cursor 0 (no bootstrap endpoint), default/max limit 200/1000, negative cursor → RFC 9457 ProblemDetail 400. Auth identical to sibling owner-scoped endpoints (`/api/**` authenticated + `media:read` API-credential scope, owner from session). - **Epoch**: clear-library bumps `sync_epoch` in the same transaction as the hard DELETE (the only signal that feed history was truncated). DB restore = operator adds 1 (ADR 0117). - **Contract**: path + `SyncChangesPage`/`SyncChange` schemas added to `specs/001-v1-baseline/contracts/core-api.openapi.yaml` following the file's existing style. ## How commit-ordered seq allocation is guaranteed (ADR 0117 risk #1) ONE coherent mechanism (documented in the migration header): a `BEFORE INSERT OR UPDATE OF <sync-visible columns>` trigger on `media_items` allocates `change_seq` from the single-row `sync_seq` counter via an atomic `INSERT … ON CONFLICT DO UPDATE … RETURNING`. The counter **row lock is held to commit**, so allocation order == commit order — the reason a plain Postgres `SEQUENCE` was rejected (non-transactional `nextval()` can commit out of order and burn gaps, which would make a client's `> cursor` keyset skip a committed change forever). The trigger deliberately covers **INSERT too** (an update-only trigger would never fire for new rows), so every writer goes through one code path. Explicit seq assignments (the backfill) pass through via an `IS DISTINCT FROM OLD` guard — a bare `IS NOT NULL` check would silently swallow every UPDATE because `NEW.change_seq` carries the row's previous value. The allocator self-heals an empty counter (DB restore) instead of failing with a not-null violation. ## Writer inventory (verified exhaustively; every writer hits the trigger) | # | Writer | Statement | File:line (main) | |---|--------|-----------|------------------| | 1 | Upload commit | INSERT INTO media_items | `apps/api/src/main/java/dev/memories/api/upload/UploadCommitService.java:280` | | 2 | Trash | UPDATE lifecycle → trashed | `apps/api/src/main/java/dev/memories/api/media/TrashService.java:48` | | 3 | Restore from trash | UPDATE lifecycle → active | `apps/api/src/main/java/dev/memories/api/media/TrashService.java:68` | | 4 | Permanent delete (single) | UPDATE lifecycle → permanently_deleted | `apps/api/src/main/java/dev/memories/api/media/TrashService.java:92` | | 5 | Empty trash (bulk purge) | UPDATE lifecycle → permanently_deleted (set-based) | `apps/api/src/main/java/dev/memories/api/media/TrashService.java:134` | | 6 | Worker metadata acceptance | UPDATE sync-visible columns (capture/width/height/location/status) | `apps/api/src/main/java/dev/memories/api/worker/WorkerResultService.java:807` (`acceptMetadataCandidate`) | | 7 | Clear library | hard DELETE FROM media_items + **epoch bump** (same tx) | `apps/api/src/main/java/dev/memories/api/media/MediaMaintenanceService.java:1378` (delete) / `:1397` (epoch) | Grep sweep over `apps/api` (all `UPDATE|INSERT INTO|DELETE FROM media_items` occurrences incl. multi-line text blocks) and `apps/worker-media` (no direct DB writes) found no other writer. Non-sync-visible UPDATEs (none exist today besides maintenance deletes) would not fire the trigger by design — the watched-column list matches the S2 payload; `updated_at` is watched, so any future service write that touches a sync field via `updated_at` is covered as a safety net. ## Tests (SyncChangeFeedIntegrationTest — real Flyway chain, Testcontainers Postgres, TrashServiceTest-style harness) - **(a)** create→trash→restore→purge: strictly increasing seqs, correct lifecycle states, single latest-state entry on replay, physical row preserved. - **(b)** keyset pagination across limit boundaries (7 items / limit 3 → 3 pages, no dup/gap, sorted, epoch on every page, exhausted-cursor pull stable) + **stability under a concurrent new change** (late row lands strictly above the taken cursor). - **(c)** owner scoping: A never sees B's item or B's trash transition. - **(d) writer-inventory test** (the ADR-demanded one): exercises writers 1–6 above end-to-end through the real services and asserts a seq bump per writer; writer 7 (hard DELETE) is pinned by the epoch test since deleted rows cannot carry a seq. - **(e)** clear-library bumps owner epoch (+1), unrelated owner's epoch untouched, post-clear pull from 0 is empty. - **(f)** trash = UPDATE (same row, new seq); purge = lifecycle-state change, no physical row loss (`COUNT(*)=1` after purge). - Rollback atomicity: a failed insert allocates no seq (counter unchanged). Payload fidelity (all S2 fields incl. contentHash from stored_objects). Negative cursor → 400. ## Validation - `rtk ./gradlew :apps:api:test`: **BUILD SUCCESSFUL — 2269 tests, 0 failures, 0 errors, 9 skipped** (pre-existing conditional skips). - Note: a first run showed 2 failures in `PersonRelationServicePostgresTest.theChosenRepresentativeWinsOverTheLatestOccurrence`; reproduced identically on a pristine checkout of base e774b693 → pre-existing flake, already tracked as ISSUE-169; passes in isolation. ## Not done (per increment plan) No Android changes, no `POST /api/sync/intents` (PR-C), no event-log table, no per-device server state, no changes to existing endpoints. KNOWN_ISSUES.md untouched (no user-visible behavior ships).
feat(api): server-authoritative change feed (spec 065 PR-A)
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 2m46s
7e722a0198
Additive, read-only, unconsumed server change feed per spec 065 S2 /
ADR 0117 Option D:

- V176: media_items.change_seq BIGINT NOT NULL (backfilled in
  per-owner created_at order), single-row GLOBAL sync_seq counter
  table, per-owner media_owner_profiles.sync_epoch (starts at 1),
  keyset index (owner_id, change_seq).
- Seq allocation is ONE coherent mechanism: a BEFORE INSERT OR UPDATE
  trigger over the counter row. The counter-row lock (held to commit)
  makes allocation order == commit order (ADR 0117 risk #1) — a
  Postgres sequence would NOT (non-transactional nextval can commit
  out of order). Trigger covers INSERT and every sync-visible UPDATE;
  explicit seq assignments (the backfill) pass through via an
  IS DISTINCT FROM OLD guard (a bare IS-NOT-NULL check would swallow
  updates: NEW carries the old seq). The allocator upsert self-heals
  an empty counter (DB restore).
- GET /api/sync/changes?cursor=&limit= (owner-scoped, media:read API
  scope, ProblemDetail 400 on negative cursor): keyset pagination,
  {epoch, changes[], nextCursor, hasMore} envelope, S2 payload
  (changeSeq, mediaItemId, lifecycleState, contentHash,
  originalFilename, byteSize, mediaType, capturedAt, updatedAt),
  state-feed semantics (latest row state wins; trash/purge are
  lifecycle transitions on permanent rows, replay-from-0 always
  possible), first sync = cursor 0, default/max limit 200/1000.
- clear-library bumps sync_epoch in the same transaction as the hard
  DELETE (the only signal that feed history was truncated).
- Contract: /api/sync/changes + SyncChangesPage/SyncChange schemas in
  specs/001 core-api.openapi.yaml.
- Tests (SyncChangeFeedIntegrationTest, real Flyway chain on
  Testcontainers Postgres): lifecycle walk (create/trash/restore/
  purge strictly increasing seqs, single latest-state entry, row
  never lost), keyset pagination across limit boundaries incl.
  exhausted-cursor pull, stability under concurrent new changes,
  owner scoping, writer-inventory test hitting EVERY media_items
  writer (upload commit, worker metadata update, trash, restore,
  permanent delete, empty-trash), rollback atomicity of allocation,
  epoch bump on clear-library (+ isolation across owners), payload
  fidelity, invalid-cursor rejection, first-sync-from-0.

Validation: full :apps:api:test suite green (2269 tests, 0 failures,
9 pre-existing skips).

KNOWN_ISSUES.md: not touched (no user-visible behavior ships).
chris merged commit e0a7f0ede9 into main 2026-10-24 13:05:37 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1352
No description provided.