fix(android): bind duplicate-content rows to one server identity (sync shadow) #1359

Merged
chris merged 1 commit from fix/sync-binder-duplicates into main 2026-10-24 21:12:42 +02:00
Owner

Root cause

A device can hold several MediaStore rows with identical bytes (one image in multiple albums/folders). ChangeBinder rule 2 (hash binding, spec 065 PR-B) required a unique unbound row per content hash:

if (candidates.size != 1) continue

With N copies of one hash, candidates.size was never exactly 1, so the change bound nothing — for every copy. Server-side trash/purge verdicts therefore never reached any copy: rows stayed plain UPLOADED, the Reconciler classified them skipped (user saw "18 übersprungen/skipped"), and the spec 065 S8 library-removed card ("Bewusst nicht in Memories") never appeared. The server side was already correct: SyncChangeQueryService.PAGE_SQL carries contentHash+byteSize even for permanently_deleted tombstones, and 0.10.0 devices already cache contentHash per row.

Behavior changes (client-only)

  1. Rule 2 fans out. A change now binds to all unbound UPLOADED rows whose contentHash matches (byteSize corroborating on both sides when both known — existing degrade-to-hash-only tolerance kept). Rationale: content identity is the server identity rule (spec 001 FR-068e/FR-068b1: identical bytes never create a second active server item), so same-hash device copies are copies of one server identity — binding them all is exact, not ambiguous. claimedHashes one-claim guard kept; rule 1 (direct serverMediaItemId refresh) untouched and final (an already-bound row is never rebound to another item).
  2. Same-hash election before binding (spec 065 C5): delete-then-reupload of byte-identical content can leave two media_items rows with one hash (the upload dedupe excludes permanently_deleted twins), so one page can carry several changes per hash. The binder elects per hash: lifecycle active > trashed > permanently_deleted, tiebreak higher changeSeq; only the winner binds that hash's rows, losers bind nothing for the hash. Unknown lifecycle values rank below known ones (consistent with ServerLifecycle.SHADOWING degrading unknowns). The purge+reupload shape now correctly binds the rows to the active twin — they do not shadow, and the purged twin cannot steal the binding.
  3. RoomSyncPullStateRepository.applyPage now reports appliedChanges = changes.size instead of shadowUpdates.size, matching the interface's documented contract ("consumed change entries — the honest pull-progress metric, independent of how many rows happened to bind", already what the in-memory test double does). With fan-out, shadow writes can exceed the change count (6 changes → 18 writes), which would have overstated the pull metric.
  4. KNOWN_ISSUES.md: appended ISSUE-192 (high/resolved/area:android) documenting the bug and fix.

Propagation

No engine changes needed — once the binder shadows rows, the existing chain surfaces them: Reconciler.shadowed → BackupRunResult.shadowedCount → BackupWorker.KEY_SHADOWED → BackupStatusViewModel.shadowedCount → BackupScreen card (shadowedCount > 0 ⇒ "Bewusst nicht in Memories"). The user's exact shape (18 rows, 6 hashes, all purged) now yields shadowedCount = 18, skippedCount = 0; no existing engine test pinned skipped for a duplicate-row shape, so none had to change.

Tests

  • ChangeBinderTest reworked to 19 cases: the bug scenario (1 purged change + 3 copies ⇒ all 3 shadowed), the exact AVD shape (6 hashes × 3 rows ⇒ 18 updates over 6 server ids), size-mismatch excludes only that copy, different hashes no cross-talk, purge+active twin (active wins, rows not shadowed), trashed-vs-purged (trashed wins), changeSeq tiebreak, unknown-lifecycle ranking, already-bound row keeps its own item while the unbound copy binds the elected winner, plus all prior single-row regressions.
  • SyncDaoTest +2: fan-out through the real Room DAO (3 rows, one change, appliedChanges = 1) and twin election through RoomSyncPullStateRepository.
  • BackupEngineTest +1 end-to-end: 3 seeded copies + one purged change page ⇒ shadowedCount = 3, skippedCount = 0, all rows bound to the purged item.

Verification

rtk ./gradlew :core:test → 140 tests, 0 failures. rtk ./gradlew :app:testDebugUnitTest → 649 tests, 0 failures. INV-1 untouched (write surface stays the three shadow columns); no server changes, no spec edits, no UI changes.

## Root cause A device can hold several MediaStore rows with **identical bytes** (one image in multiple albums/folders). `ChangeBinder` rule 2 (hash binding, spec 065 PR-B) required a **unique** unbound row per content hash: ```kotlin if (candidates.size != 1) continue ``` With N copies of one hash, `candidates.size` was never exactly 1, so the change bound **nothing** — for every copy. Server-side trash/purge verdicts therefore never reached any copy: rows stayed plain `UPLOADED`, the `Reconciler` classified them `skipped` (user saw "18 übersprungen/skipped"), and the spec 065 S8 library-removed card ("Bewusst nicht in Memories") never appeared. The server side was already correct: `SyncChangeQueryService.PAGE_SQL` carries `contentHash`+`byteSize` even for `permanently_deleted` tombstones, and 0.10.0 devices already cache `contentHash` per row. ## Behavior changes (client-only) 1. **Rule 2 fans out.** A change now binds to **all** unbound `UPLOADED` rows whose `contentHash` matches (byteSize corroborating on both sides when both known — existing degrade-to-hash-only tolerance kept). Rationale: content identity *is* the server identity rule (spec 001 FR-068e/FR-068b1: identical bytes never create a second active server item), so same-hash device copies are copies of one server identity — binding them all is exact, not ambiguous. `claimedHashes` one-claim guard kept; rule 1 (direct `serverMediaItemId` refresh) untouched and final (an already-bound row is never rebound to another item). 2. **Same-hash election before binding** (spec 065 C5): delete-then-reupload of byte-identical content can leave two `media_items` rows with one hash (the upload dedupe excludes `permanently_deleted` twins), so one page can carry several changes per hash. The binder elects per hash: lifecycle `active` > `trashed` > `permanently_deleted`, tiebreak higher `changeSeq`; only the winner binds that hash's rows, losers bind nothing for the hash. Unknown lifecycle values rank below known ones (consistent with `ServerLifecycle.SHADOWING` degrading unknowns). The purge+reupload shape now correctly binds the rows to the **active** twin — they do not shadow, and the purged twin cannot steal the binding. 3. **`RoomSyncPullStateRepository.applyPage`** now reports `appliedChanges = changes.size` instead of `shadowUpdates.size`, matching the interface's documented contract ("consumed change entries — the honest pull-progress metric, independent of how many rows happened to bind", already what the in-memory test double does). With fan-out, shadow writes can exceed the change count (6 changes → 18 writes), which would have overstated the pull metric. 4. **KNOWN_ISSUES.md**: appended `ISSUE-192` (`high`/`resolved`/`area:android`) documenting the bug and fix. ## Propagation No engine changes needed — once the binder shadows rows, the existing chain surfaces them: `Reconciler.shadowed` → `BackupRunResult.shadowedCount` → `BackupWorker.KEY_SHADOWED` → `BackupStatusViewModel.shadowedCount` → `BackupScreen` card (`shadowedCount > 0` ⇒ "Bewusst nicht in Memories"). The user's exact shape (18 rows, 6 hashes, all purged) now yields `shadowedCount = 18`, `skippedCount = 0`; no existing engine test pinned `skipped` for a duplicate-row shape, so none had to change. ## Tests - `ChangeBinderTest` reworked to 19 cases: the bug scenario (1 purged change + 3 copies ⇒ all 3 shadowed), the exact AVD shape (6 hashes × 3 rows ⇒ 18 updates over 6 server ids), size-mismatch excludes only that copy, different hashes no cross-talk, purge+active twin (active wins, rows not shadowed), trashed-vs-purged (trashed wins), changeSeq tiebreak, unknown-lifecycle ranking, already-bound row keeps its own item while the unbound copy binds the elected winner, plus all prior single-row regressions. - `SyncDaoTest` +2: fan-out through the real Room DAO (3 rows, one change, `appliedChanges = 1`) and twin election through `RoomSyncPullStateRepository`. - `BackupEngineTest` +1 end-to-end: 3 seeded copies + one purged change page ⇒ `shadowedCount = 3`, `skippedCount = 0`, all rows bound to the purged item. ## Verification `rtk ./gradlew :core:test` → 140 tests, 0 failures. `rtk ./gradlew :app:testDebugUnitTest` → 649 tests, 0 failures. INV-1 untouched (write surface stays the three shadow columns); no server changes, no spec edits, no UI changes.
fix(android): bind duplicate-content rows to one server identity (sync shadow)
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Successful in 3m38s
03619450d2
ChangeBinder rule 2 demanded a UNIQUE unbound row per content hash, so a
device holding several copies of one image (same SHA-256) never bound:
server trash/purge verdicts never reached any copy, rows stayed plain
UPLOADED and the reconciler reported them as skipped — the spec 065 S8
library-removed card ('Bewusst nicht in Memories') never appeared and
the deletion was invisible on the device.

- Rule 2 now fans out: ALL unbound UPLOADED rows matching
  (contentHash, byteSize-corroborated) bind to the change — content
  identity is the server's identity rule (spec 001 FR-068e/FR-068b1
  dedupe), so same-hash device copies are one server identity, not an
  ambiguity. claimedHashes one-claim guard kept; rule 1 (direct
  serverMediaItemId refresh) untouched and final (no rebinding).
- Same-hash election per page (spec 065 C5 purge+reupload twins):
  lifecycle active > trashed > permanently_deleted, tiebreak higher
  changeSeq; only the winning change binds the hash's rows. Unknown
  lifecycle values rank below known ones.
- Room applyPage now reports appliedChanges = changes.size per the
  interface contract (binding is fan-out; shadow writes overstate the
  pull metric — 6 changes could count as 18).
- KNOWN_ISSUES.md: ISSUE-192 (resolved).

Tests: ChangeBinderTest reworked (19 cases: AVD 18-rows/6-hashes shape,
purge+active twin, trashed-vs-purged, size-mismatch exclusion, bound-row
no-rebind, regressions); SyncDaoTest +2 (Room fan-out + twin election
through the real DAO); BackupEngineTest +1 (3 copies shadow, 0 skipped).
chris merged commit 78484efeae into main 2026-10-24 21:12:42 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1359
No description provided.