fix(backup): deduped uploads counted as skipped, not uploaded #1385

Merged
chris merged 1 commit from fix/upload-dedup-count into main 2026-10-25 19:00:25 +01:00
Owner

0.10.10 report: uploading 3×6 byte-identical photos showed „18 hochgeladen". The server already dedupes at commit (S4-C1 `skipExistingOwnerDuplicate` → `deduplication_state = skipped_existing_owner_media`, storage reuse `reused_stored_object`), but the commit response carried no per-file verdict, so the client counted every file entering the session as uploaded.

Server (additive only)

  • Commit response now carries `files: [{fileId, deduplicationState}]` for exactly the files the call committed (`UploadSessionFileDedup`, `@JsonInclude(NON_NULL)` — the field is absent on every aggregate `UploadSession` read, so older clients/web/CLI ignore it).
  • `specs/001-v1-baseline/contracts/core-api.openapi.yaml`: `UploadSession.files` + `UploadSessionFileDedup` schema, commit 202 description amended.

Client

  • `UploadSessionClient` parses `files` (default empty → older servers behave exactly as before).
  • Ruling: `skipped_existing_owner_media` = DEDUPED (owner already had this media); `reused_stored_object` and `new_stored_object` = UPLOADED (a new media item was created; the former only reuses stored bytes).
  • `BackupRunResult.dedupedCount` (default 0 = source-compatible); `uploadedCount` counts new items only; `skippedCount` keeps its reconciler meaning — deduped is its own counter, never merged.
  • INVARIANT kept: deduped files are still marked UPLOADED locally (they ARE on the server) — only the run statistics change.
  • `BackupWorker` output gains `dedupedCount` (own bucket, null-safe on pre-fix run outputs); Backup stats row appends „Duplikate übersprungen N" / "duplicates skipped N" / "dubletter sprunget over N" — 3×6 upload now reads „hochgeladen 6 · … · Duplikate übersprungen 12".
  • `BackupRunProgress.Committing` left untouched: both consumers ignore it (worker → `Unit`) and it fires before a response exists.

Tests — server: commit response carries per-file verdicts (reused + skipped cases; aggregate reads omit the field); client: engine 1 new + 2 deduped → uploadedCount=1, dedupedCount=2, all 3 UPLOADED; reused_stored_object counts as uploaded; absent-verdict degradation; response-parse contract; worker snapshot key contract + pre-fix degradation; stats formatter incl. de/en/da.

Verification

  • `rtk ./gradlew :apps:api:test` — UploadCommitServiceTest 11/11 green; full run green except the ISSUE-169 load-dependent flake `PersonRelationServicePostgresTest.theChosenRepresentativeWinsOverTheLatestOccurrence` (thumbnail ladder; zero code-path overlap; passes 3/3 with `--rerun` on this branch).
  • `rtk ./gradlew :core:test` — 176 tests, 0 failures.
  • `rtk ./gradlew :app:testDebugUnitTest` — 712 tests, 0 failures.

KNOWN_ISSUES: ISSUE-206 (med/resolved/area:api). Rebased on origin/main eb63219f. Do NOT merge — orchestrator reviews.

0.10.10 report: uploading 3×6 byte-identical photos showed „18 hochgeladen". The server already dedupes at commit (S4-C1 \`skipExistingOwnerDuplicate\` → \`deduplication_state = skipped_existing_owner_media\`, storage reuse \`reused_stored_object\`), but the commit response carried no per-file verdict, so the client counted every file entering the session as uploaded. **Server (additive only)** - Commit response now carries \`files: [{fileId, deduplicationState}]\` for exactly the files the call committed (\`UploadSessionFileDedup\`, \`@JsonInclude(NON_NULL)\` — the field is absent on every aggregate \`UploadSession\` read, so older clients/web/CLI ignore it). - \`specs/001-v1-baseline/contracts/core-api.openapi.yaml\`: \`UploadSession.files\` + \`UploadSessionFileDedup\` schema, commit 202 description amended. **Client** - \`UploadSessionClient\` parses \`files\` (default empty → older servers behave exactly as before). - **Ruling:** \`skipped_existing_owner_media\` = DEDUPED (owner already had this media); \`reused_stored_object\` and \`new_stored_object\` = UPLOADED (a new media item was created; the former only reuses stored bytes). - \`BackupRunResult.dedupedCount\` (default 0 = source-compatible); \`uploadedCount\` counts new items only; \`skippedCount\` keeps its reconciler meaning — deduped is its own counter, never merged. - INVARIANT kept: deduped files are still marked UPLOADED locally (they ARE on the server) — only the run statistics change. - \`BackupWorker\` output gains \`dedupedCount\` (own bucket, null-safe on pre-fix run outputs); Backup stats row appends „Duplikate übersprungen N" / "duplicates skipped N" / "dubletter sprunget over N" — 3×6 upload now reads „hochgeladen 6 · … · Duplikate übersprungen 12". - \`BackupRunProgress.Committing\` left untouched: both consumers ignore it (worker → \`Unit\`) and it fires before a response exists. **Tests** — server: commit response carries per-file verdicts (reused + skipped cases; aggregate reads omit the field); client: engine 1 new + 2 deduped → uploadedCount=1, dedupedCount=2, all 3 UPLOADED; reused_stored_object counts as uploaded; absent-verdict degradation; response-parse contract; worker snapshot key contract + pre-fix degradation; stats formatter incl. de/en/da. **Verification** - \`rtk ./gradlew :apps:api:test\` — UploadCommitServiceTest 11/11 green; full run green except the ISSUE-169 load-dependent flake \`PersonRelationServicePostgresTest.theChosenRepresentativeWinsOverTheLatestOccurrence\` (thumbnail ladder; zero code-path overlap; passes 3/3 with \`--rerun\` on this branch). - \`rtk ./gradlew :core:test\` — 176 tests, 0 failures. - \`rtk ./gradlew :app:testDebugUnitTest\` — 712 tests, 0 failures. KNOWN_ISSUES: ISSUE-206 (med/resolved/area:api). Rebased on origin/main eb63219f. Do NOT merge — orchestrator reviews.
feat(backup): report server-side dedupe in upload stats
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 9m41s
Build and Deploy / Android unit tests and debug APK (push) Successful in 3m59s
2c996a26f5
0.10.10 counted server-side dedupes as uploaded (3x6 identical photos
showed '18 hochgeladen'): the commit response carried no per-file dedup
verdict, so the client counted every file entering the session as
uploaded even though UploadCommitService had set deduplication_state and
skipped creating the item.

Server (additive): the commit response now carries
files:[{fileId,deduplicationState}] for exactly the files the call
committed (UploadSessionFileDedup, NON_NULL - absent on every aggregate
read so older clients are unaffected); OpenAPI UploadSession amended.

Client: BackupEngine parses the verdicts and adds
BackupRunResult.dedupedCount. Ruling: skipped_existing_owner_media =
deduped (owner already had the media); reused_stored_object /
new_stored_object = uploaded (new media item, bytes possibly shared).
Deduped files STAY marked UPLOADED locally - only the run statistics
split them out of uploadedCount; skippedCount keeps its reconciler
meaning. BackupWorker/BackupWorkSnapshot carry dedupedCount as their own
bucket and the stats row appends 'Duplikate übersprungen N' /
'duplicates skipped N' / 'dubletter sprunget over N' (de/en/da).

KNOWN_ISSUES: ISSUE-206 (med/resolved/area:api).
chris merged commit f145f836fb into main 2026-10-25 19:00:25 +01:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1385
No description provided.