fix(android): percent-encode upload filename headers + per-item failure isolation (ISSUE-233) #1523

Merged
chris merged 3 commits from fix/android-upload-filename-peritem into main 2026-11-15 17:47:01 +01:00
Owner

Problem (user report, production)

Uploading a file named "Fødselsdagsinvitation med sportsmotiv3.png" from Android crashed twice over:

  1. IllegalArgumentException: Unexpected char 0xf8 at 1 in filename value — UploadSessionClient sent filename / client-relative-path RAW in HTTP headers; OkHttp's header-value check rejects non-Latin-1 chars before any bytes are sent.
  2. Whole-session abort: the exception escaped the backup engine's batch loop, so the session was never committed and the files already RECEIVED (~404 items) were stranded unprocessed.

Fix (client-side only)

1. Percent-encode the fingerprint headers (UploadSessionClient)

  • filename + client-relative-path now encoded at the header boundary with encodeURIComponent semantics (RFC 3986 UTF-8, space → %20) — mirrors the web client's wire format (apps/web/src/lib/api/upload.ts). The server already percent-decodes these once for display only (UploadActivityService.decodeOnceForDisplay), so Android was the wire-format outlier.
  • client-file-key stays RAW: web sends it raw and the server keys upload_session_files rows on it verbatim (upsert ON CONFLICT (upload_session_id, client_file_key)); Android keys are ASCII by construction (mediastore:<id>, picker:<uri>).
  • FileUpload fields stay raw — encoding is wire-only, display/path logic keeps the real name.

2. Per-item failure isolation (BackupEngine.run)

  • Faults local to one item (request-build errors, unreadable MediaStore stream, hashing I/O) now mark that item FAILED (existing per-file state; retryable next run via reconcile) and the loop continues.
  • The session is still committed with the successfully received files — the server's finalizing commit processes the full received set.
  • Abort semantics preserved: 401 rethrows; 403/404/409/410 → batch discarded + abort; 5xx/network (after retry budget) → batch kept + abort; CancellationException never swallowed.

Server: verified already-safe, deliberately unchanged

  • Finalizing commit (fileIds == null) commits all received/classified/commit_pending files (UploadCommitService.eligibleFiles) — the commit set is the received set.
  • Per-file commit failures abort via transaction rollback (staging kept, idempotent replay with the persisted key).
  • A session that never commits is the owner's discard/restore decision under spec 070 (stale transition, merged as PR #1519). Enqueueing at receive time or auto-processing stale sessions would contradict its model.
  • ISSUE-210 receive lifecycle (no transaction during body streaming) and SEC-4 ownership checks untouched.

Tests

  • :core:test — 207 tests, 0 failures (Android :core only; no server code touched).
    • UploadSessionClientTest: Danish filename sent as F%C3%B8dselsdagsinvitation%20med%20sportsmotiv3.png; ASCII client-file-key passthrough; DTO keeps raw name.
    • BackupEngineTest: local read failure isolates per item + batch still commits; failed item retries in a fresh session next run.

Known issues

  • KNOWN_ISSUES.md: [ISSUE-233] high resolved area:upload (228 was taken; 233 next free).

Scope

  • No changes to TimelineScreen.kt / TimelineViewModel. Server code untouched.
## Problem (user report, production) Uploading a file named **"Fødselsdagsinvitation med sportsmotiv3.png"** from Android crashed twice over: 1. **IllegalArgumentException: Unexpected char 0xf8 at 1 in filename value** — `UploadSessionClient` sent `filename` / `client-relative-path` RAW in HTTP headers; OkHttp's header-value check rejects non-Latin-1 chars before any bytes are sent. 2. **Whole-session abort**: the exception escaped the backup engine's batch loop, so the session was never committed and the files already RECEIVED (~404 items) were stranded unprocessed. ## Fix (client-side only) ### 1. Percent-encode the fingerprint headers (`UploadSessionClient`) - `filename` + `client-relative-path` now encoded at the header boundary with `encodeURIComponent` semantics (RFC 3986 UTF-8, space → `%20`) — mirrors the web client's wire format (`apps/web/src/lib/api/upload.ts`). The server already percent-decodes these once for display only (`UploadActivityService.decodeOnceForDisplay`), so Android was the wire-format outlier. - `client-file-key` stays **RAW**: web sends it raw and the server keys `upload_session_files` rows on it verbatim (upsert `ON CONFLICT (upload_session_id, client_file_key)`); Android keys are ASCII by construction (`mediastore:<id>`, `picker:<uri>`). - `FileUpload` fields stay raw — encoding is wire-only, display/path logic keeps the real name. ### 2. Per-item failure isolation (`BackupEngine.run`) - Faults **local** to one item (request-build errors, unreadable MediaStore stream, hashing I/O) now mark that item **FAILED** (existing per-file state; retryable next run via reconcile) and the loop continues. - The session is still **committed** with the successfully received files — the server's finalizing commit processes the full received set. - Abort semantics preserved: 401 rethrows; 403/404/409/410 → batch discarded + abort; 5xx/network (after retry budget) → batch kept + abort; `CancellationException` never swallowed. ### Server: verified already-safe, deliberately unchanged - Finalizing commit (`fileIds == null`) commits **all** `received/classified/commit_pending` files (`UploadCommitService.eligibleFiles`) — the commit set is the received set. - Per-file commit failures abort via **transaction rollback** (staging kept, idempotent replay with the persisted key). - A session that never commits is the owner's discard/restore decision under **spec 070** (stale transition, merged as PR #1519). Enqueueing at receive time or auto-processing stale sessions would contradict its model. - ISSUE-210 receive lifecycle (no transaction during body streaming) and SEC-4 ownership checks untouched. ## Tests - `:core:test` — **207 tests, 0 failures** (Android `:core` only; no server code touched). - `UploadSessionClientTest`: Danish filename sent as `F%C3%B8dselsdagsinvitation%20med%20sportsmotiv3.png`; ASCII `client-file-key` passthrough; DTO keeps raw name. - `BackupEngineTest`: local read failure isolates per item + batch still commits; failed item retries in a fresh session next run. ## Known issues - `KNOWN_ISSUES.md`: `[ISSUE-233]` `high` `resolved` `area:upload` (228 was taken; 233 next free). ## Scope - No changes to TimelineScreen.kt / TimelineViewModel. Server code untouched.
Production crash: uploading a file named "Fødselsdagsinvitation med
sportsmotiv3.png" from Android threw
IllegalArgumentException: Unexpected char 0xf8 at 1 in filename value —
OkHttp's header-value check rejects raw non-Latin-1 header values, so the
request was never sent and the exception escaped the engine's batch loop.

UploadSessionClient.uploadFile now percent-encodes the `filename` and
`client-relative-path` headers at the header boundary with
encodeURIComponent semantics (RFC 3986 UTF-8, space -> %20), mirroring
the web client's wire format in apps/web/src/lib/api/upload.ts. The
server percent-decodes these headers exactly once for display only
(UploadActivityService#decodeOnceForDisplay), so the on-wire values were
already specified as percent-encoded — Android was the outlier.

`client-file-key` stays RAW on purpose: web sends it raw and the server
keys upload_session_files rows on it verbatim (upsert ON CONFLICT
(upload_session_id, client_file_key)); Android keys are ASCII by
construction (mediastore:<id>, picker:<uri>).

FileUpload fields stay RAW: encoding is a wire detail of the client, so
display/path logic keeps the real name. Tests pin the Danish-filename
encoding contract, the ASCII key passthrough, and the raw DTO flow.
Second half of the production incident (same report): with a Danish-named
file in the batch, the whole upload session aborted even though earlier
files were already received — the exception escaped the engine's batch
loop, so the commit never ran and every already-received file of the
session stayed unprocessed.

BackupEngine.run's batch loop now catches faults LOCAL to one item
(request-build errors and local IO faults such as an unreadable
MediaStore stream / hashing I/O) after the session-level catches:
- the item is marked FAILED with the fault in failReason (existing
  per-file state machinery; FAILED reconciles back to pending with its
  cached hash on the next run, so nothing is silently lost),
- the loop continues with the remaining items,
- the session is still committed with the successfully received files —
  the finalizing commit processes the server's full received set.

Preserved abort semantics, in order: AuthExpiredException rethrows;
SessionGoneException discards the batch and aborts; 5xx/network
(UploadAbortedException, after the in-run retry budget) aborts with the
batch kept — continuing would only fail the commit against the same dead
transport. CancellationException is rethrown and never swallowed, so a
WorkManager stop still cancels the run.

Tests: local read failure isolates per item and the batch still commits
(no file request for the failed item); a batch with a failed item commits
received files and the failed item retries successfully in a fresh
session on the next run. The 400-rejection isolation and 5xx-abort tests
are untouched and stay green.
docs(known-issues): ISSUE-233 android non-Latin-1 upload filename crash + whole-batch abort (resolved)
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Successful in 5m55s
409cca549b
Records the production incident and the resolution shipped on
fix/android-upload-filename-peritem: percent-encoded fingerprint headers
at the client boundary plus per-item failure isolation in the backup
engine. Also documents the evidence-backed decision NOT to change the
server: the finalizing commit already processes the entire received set
(UploadCommitService.eligibleFiles with fileIds == null), per-file commit
failures roll back via the transaction with staging kept for replay, and
a session that never commits is the owner's discard/restore decision
under spec 070 (stale) — receive stays transactionless (ISSUE-210) and
SEC-4 ownership checks are untouched.
chris force-pushed fix/android-upload-filename-peritem from 409cca549b
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Successful in 5m55s
to 08a7bc9f78
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build Android CI image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Deploy review instance (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 4s
Build and Deploy / Android unit tests and debug APK (push) Successful in 5m15s
2026-11-15 17:40:48 +01:00
Compare
chris merged commit c082c903db into main 2026-11-15 17:47:01 +01:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1523
No description provided.