fix(android-core): recover stale upload batches, serialize runs, retry transient 5xx #1341

Merged
chris merged 1 commit from fix/upload-engine into main 2026-10-23 22:10:09 +02:00
Owner

Three correctness fixes in the Android upload engine (:core only — apps/android/app, apps/web, apps/api untouched).

1. Session lifecycle (stale-batch recovery)

The engine reused a persisted open batch (upload_batch) with no validation. When the server answered 403 Not the owner of this upload session (session created under the API credential but presented with cookie auth, or vice versa), the run aborted, the batch survived restart, and every retry looped on the same rejected session forever.

  • On run start with an open batch: GET /api/upload-sessions/{id} probe.
  • 403/404/409/410 (all SessionGoneException now; 403 newly mapped in UploadSessionClient) or terminal status (completed*/cancelled/failed/expired) → discard batch, create a fresh session.
  • Live status → reuse with the original idempotency key (crash-retry replay preserved; re-sent file uploads dedupe server-side via upsert on (upload_session_id, client_file_key)).
  • Client-side recovery only, per SEC-4 — ownership is never "fixed" server-side.

2. Single-flight guard

Periodic (memories-backup-periodic) and immediate (memories-backup-now) work have distinct unique names and could run concurrently in one process, streaming the same files into the same session (observed interleaved duplicates).

  • BackupRunGate: process-wide Mutex on a module-level singleton, acquired by BackupEngine.run.
  • Wait, not skip (rationale in the gate KDoc): a skip would silently drop the periodic run's work until the next 6 h tick, while the persisted batch makes a waiter's run a correct incremental follow-up — WorkManager retry/idempotency stays sane by construction. Cross-process serialization remains WorkManager's job and is out of scope.

3. Transient 5xx retry

A single 502 from the reverse proxy aborted the whole run; WorkManager then restarted it from scratch every ~30 s against a struggling server.

  • UploadRetryPolicy (4 retries, 2 s → 60 s cap, ±25% jitter) around file upload and commit.
  • Never retried in-run: 4xx rejections, session-gone family, auth failures. Budget exhausted → abort → WorkManager retry as before.
  • Engine tests default to UploadRetryPolicy.NONE so the pre-existing abort-semantics contracts stay pinned byte-for-byte.

Behavior changes to existing tests (deliberate)

Test Change
crash retry reuses the session… run 2 now expects a GET probe (revalidation) before reuse — same session + key, still exactly one create
5xx aborts the run… unchanged in intent; now guaranteed no in-run retries via the NONE policy so request counts stay exact
session gone clears batch… unchanged (404) — 403 now covered by two new dedicated tests

Tests

cd apps/android && ./gradlew :core:test
UploadRetryPolicyTest: 4, ManualUploadCoordinatorTest: 6, UploadSessionClientTest: 16,
BackupEngineTest: 24, ContentHasherTest: 3, NetworkClientTest: 2, ReconcilerTest: 11
TOTAL: 66/66 passed (also verified with --rerun-tasks)

Environment limitation: no Android SDK on this box, so :app could not be compiled locally. The change is source-compatible by inspection: AndroidSyncController uses named constructor args that all still exist; the two new engine params (retryPolicy, retrySleeper) have defaults. Please let CI's android job (:app:testDebugUnitTest, :app:assembleDebug) confirm.

KNOWN_ISSUES.md: ISSUE-176/177/178 appended (mitigated, ref fix/upload-engine).

Three correctness fixes in the Android upload engine (`:core` only — `apps/android/app`, `apps/web`, `apps/api` untouched). ## 1. Session lifecycle (stale-batch recovery) The engine reused a persisted open batch (`upload_batch`) with **no validation**. When the server answered 403 `Not the owner of this upload session` (session created under the API credential but presented with cookie auth, or vice versa), the run aborted, the batch survived restart, and every retry looped on the same rejected session forever. - On run start with an open batch: `GET /api/upload-sessions/{id}` probe. - **403/404/409/410** (all `SessionGoneException` now; 403 newly mapped in `UploadSessionClient`) or **terminal status** (`completed*`/`cancelled`/`failed`/`expired`) → discard batch, create a fresh session. - **Live status** → reuse with the original idempotency key (crash-retry replay preserved; re-sent file uploads dedupe server-side via upsert on `(upload_session_id, client_file_key)`). - Client-side recovery only, per SEC-4 — ownership is never "fixed" server-side. ## 2. Single-flight guard Periodic (`memories-backup-periodic`) and immediate (`memories-backup-now`) work have distinct unique names and could run concurrently in one process, streaming the same files into the same session (observed interleaved duplicates). - `BackupRunGate`: process-wide `Mutex` on a module-level singleton, acquired by `BackupEngine.run`. - **Wait, not skip** (rationale in the gate KDoc): a skip would silently drop the periodic run's work until the next 6 h tick, while the persisted batch makes a waiter's run a correct incremental follow-up — WorkManager retry/idempotency stays sane by construction. Cross-process serialization remains WorkManager's job and is out of scope. ## 3. Transient 5xx retry A single 502 from the reverse proxy aborted the whole run; WorkManager then restarted it from scratch every ~30 s against a struggling server. - `UploadRetryPolicy` (4 retries, 2 s → 60 s cap, ±25% jitter) around file upload and commit. - **Never retried in-run:** 4xx rejections, session-gone family, auth failures. Budget exhausted → abort → WorkManager retry as before. - Engine tests default to `UploadRetryPolicy.NONE` so the pre-existing abort-semantics contracts stay pinned byte-for-byte. ## Behavior changes to existing tests (deliberate) | Test | Change | |---|---| | `crash retry reuses the session…` | run 2 now expects a `GET` probe (revalidation) before reuse — same session + key, still exactly one create | | `5xx aborts the run…` | unchanged in intent; now guaranteed no in-run retries via the `NONE` policy so request counts stay exact | | `session gone clears batch…` | unchanged (404) — 403 now covered by two new dedicated tests | ## Tests ``` cd apps/android && ./gradlew :core:test UploadRetryPolicyTest: 4, ManualUploadCoordinatorTest: 6, UploadSessionClientTest: 16, BackupEngineTest: 24, ContentHasherTest: 3, NetworkClientTest: 2, ReconcilerTest: 11 TOTAL: 66/66 passed (also verified with --rerun-tasks) ``` **Environment limitation:** no Android SDK on this box, so `:app` could not be compiled locally. The change is source-compatible by inspection: `AndroidSyncController` uses named constructor args that all still exist; the two new engine params (`retryPolicy`, `retrySleeper`) have defaults. Please let CI's android job (`:app:testDebugUnitTest`, `:app:assembleDebug`) confirm. KNOWN_ISSUES.md: ISSUE-176/177/178 appended (`mitigated`, ref `fix/upload-engine`).
fix(android-core): recover stale upload batches, serialize runs, retry transient 5xx
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Successful in 3m14s
c477e8d723
Three correctness fixes in the :core upload engine (apps/android/core only):

1. Session lifecycle: an open batch is validated via GET
   /api/upload-sessions/{id} before reuse. 403/404/409/410 (now all
   SessionGoneException — 403 covers the server's 'Not the owner of this
   upload session' credential-mismatch case) and terminal statuses
   (completed*/cancelled/failed/expired) discard the batch so a fresh
   session is created; a live session is reused with its original
   idempotency key, preserving crash-retry replay. Client-side recovery
   only, per SEC-4 — ownership is never 'fixed' server-side.

2. Single-flight guard: BackupRunGate (process-wide Mutex on a module
   singleton) serializes every BackupEngine.run in one process. Wait-not-
   skip semantics: the second run queues and then proceeds incrementally;
   a skip would drop the periodic run's work until the next 6h tick,
   while the persisted batch keeps a waiter's run idempotent.

3. Transient retry: bounded exponential backoff with jitter
   (UploadRetryPolicy: 4 retries, 2s->60s cap, ±25%) around file upload
   and commit for 5xx/network faults before falling back to abort ->
   WorkManager retry. 4xx (except the session-gone family), auth and
   session-gone failures are never retried in-run. Replayed file uploads
   are server-side idempotent (upsert on upload_session_id+client_file_key).

Tests: BackupEngineTest 14→24 (probe-before-reuse, 403-upload/commit
clearing the batch, stale-batch recovery for rejected and terminal
sessions, commit/file 5xx in-run retry incl. budget exhaustion, 4xx
never retried, gate serialization), UploadSessionClientTest +3 (403
mappings), UploadRetryPolicyTest (new, backoff math). Existing tests
updated for the new behavior: crash-retry and fresh-batch runs now
expect a GET probe; BatchEngine default test policy disables in-run
retry so abort semantics stay pinned.

KNOWN_ISSUES.md: append ISSUE-176/177/178 (mitigated, fix/upload-engine).
chris force-pushed fix/upload-engine from c477e8d723
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Successful in 3m14s
to 10805a3aa1
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 3s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 4s
Build and Deploy / Android unit tests and debug APK (push) Successful in 3m59s
2026-10-23 22:05:48 +02:00
Compare
chris merged commit 262f518573 into main 2026-10-23 22:10:09 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1341
No description provided.