fix(media): decode URL-encoded filenames and guard capture-year bounds #1028

Merged
chris merged 1 commit from fix/capture-year-guard into main 2026-10-04 10:31:45 +02:00
Owner

Root cause

Ingested original filenames are stored percent-encoded in original_assets.original_filename (e.g. video_87%4001-03-2019_10-15-42.mp4 for real name video_87@01-03-2019_10-15-42.mp4). MediaJobHandlers.timestampFromFilename matched its YYYY-MM-DD patterns against the encoded string, so %40+01 read as year 4001 (greedy (\\d{4})), shifting month/day/hour too: capture_timestamp = 4001-03-20 19:10:15 with confidence inferred. No year sanity check existed anywhere on the write path, so the start page rendered timeline sections like "March 4001" (21 active media_items in prod).

Fix (write side only; read queries and frontend untouched)

  • worker-media: percent-decode the filename (UTF-8 %HH only, literal + preserved, malformed escapes fall back to the raw name) before applying the filename-date patterns — used for date extraction only, never for storage/echo. video_87%4001-... now decodes to video_87@01-03-2019_... and matches no pattern → item stays undated (correct outcome).
  • worker-media: year guard on the final captureTimestamp in extractMetadata for ALL sources (EXIF DateTimeOriginal, DateTimeDigitized, filename-derived): year < 1800 or > current year + 1 (worker zone) → value omitted from the emitted metadata map.
  • api: WorkerResultService.acceptMetadataCandidate treats an incoming captureTimestamp with an implausible year as absent — timestamp, timezone and confidence are not bound, so COALESCE preserves the stored values and the row never claims a timestamp it did not write.

Tiny isPlausibleCaptureYear helper duplicated per module (no shared module exists; no new module wiring).

Tests

  • MediaJobHandlersTest: encoded video_87%4001-03-2019_10-15-42.mp4 → no capture timestamp (regression); clip%402019-03-01T10-15-42.mp4 → decodes and yields 2019-03-01T10:15:42Z; video_4001-01-01_10-00-00.mp4 → rejected by guard; existing sane-date tests unchanged.
  • MLResultAcceptanceTest: candidate with captureTimestamp year 4001 leaves an existing capture_timestamp/capture_timezone/timestamp_confidence untouched, and leaves a fresh item undated; sane timestamps still write (existing tests).

rtk ./gradlew :apps:worker-media:test → 29 tests, 0 failures.
rtk ./gradlew :apps:api:test → 1546 tests, 0 failures, 9 skipped (pre-existing assumptions).

Refs: ISSUE-098 (KNOWN_ISSUES.md). Existing prod rows are repaired separately.

## Root cause Ingested original filenames are stored percent-encoded in `original_assets.original_filename` (e.g. `video_87%4001-03-2019_10-15-42.mp4` for real name `video_87@01-03-2019_10-15-42.mp4`). `MediaJobHandlers.timestampFromFilename` matched its YYYY-MM-DD patterns against the **encoded** string, so `%40`+`01` read as year **4001** (greedy `(\\d{4})`), shifting month/day/hour too: `capture_timestamp = 4001-03-20 19:10:15` with confidence `inferred`. No year sanity check existed anywhere on the write path, so the start page rendered timeline sections like "March 4001" (21 active media_items in prod). ## Fix (write side only; read queries and frontend untouched) - **worker-media**: percent-decode the filename (UTF-8 `%HH` only, literal `+` preserved, malformed escapes fall back to the raw name) before applying the filename-date patterns — used for date extraction only, never for storage/echo. `video_87%4001-...` now decodes to `video_87@01-03-2019_...` and matches no pattern → item stays undated (correct outcome). - **worker-media**: year guard on the final `captureTimestamp` in `extractMetadata` for ALL sources (EXIF DateTimeOriginal, DateTimeDigitized, filename-derived): year < 1800 or > current year + 1 (worker zone) → value omitted from the emitted metadata map. - **api**: `WorkerResultService.acceptMetadataCandidate` treats an incoming `captureTimestamp` with an implausible year as absent — timestamp, timezone and confidence are not bound, so `COALESCE` preserves the stored values and the row never claims a timestamp it did not write. Tiny `isPlausibleCaptureYear` helper duplicated per module (no shared module exists; no new module wiring). ## Tests - `MediaJobHandlersTest`: encoded `video_87%4001-03-2019_10-15-42.mp4` → no capture timestamp (regression); `clip%402019-03-01T10-15-42.mp4` → decodes and yields `2019-03-01T10:15:42Z`; `video_4001-01-01_10-00-00.mp4` → rejected by guard; existing sane-date tests unchanged. - `MLResultAcceptanceTest`: candidate with `captureTimestamp` year 4001 leaves an existing `capture_timestamp`/`capture_timezone`/`timestamp_confidence` untouched, and leaves a fresh item undated; sane timestamps still write (existing tests). `rtk ./gradlew :apps:worker-media:test` → 29 tests, 0 failures. `rtk ./gradlew :apps:api:test` → 1546 tests, 0 failures, 9 skipped (pre-existing assumptions). Refs: ISSUE-098 (KNOWN_ISSUES.md). Existing prod rows are repaired separately.
fix(media): decode URL-encoded filenames and guard capture-year bounds
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 1m44s
05515bc59c
Ingested original filenames are stored percent-encoded in
original_assets.original_filename (e.g. 'video_87%4001-03-2019_10-15-42.mp4'
for the real name 'video_87@01-03-2019_10-15-42.mp4'). The media worker's
timestampFromFilename matched its YYYY-MM-DD patterns against that encoded
string, so the '%40'+'01' sequence read as year 4001 (greedy (\d{4})),
shifting month/day/hour as well: capture_timestamp 4001-03-20 19:10:15 with
confidence 'inferred'. No year sanity check existed anywhere on the write
path, so the web start page rendered timeline sections titled 'March 4001'
(21 active media_items affected in prod).

Three-part write-side fix:
- worker: percent-decode the filename (UTF-8 %HH only, literal '+' preserved,
  malformed escapes fall back to the raw name) before applying the
  filename-date patterns; the decoded value is used only for date extraction,
  so 'video_87%4001-...' now stays undated instead of yielding year 4001.
- worker: guard the final captureTimestamp in extractMetadata for ALL sources
  (EXIF DateTimeOriginal, DateTimeDigitized, filename-derived); values with
  year < 1800 or > current year + 1 (worker zone) are omitted from the
  emitted metadata map.
- api: WorkerResultService.acceptMetadataCandidate treats an incoming
  captureTimestamp with an implausible year as absent (timestamp, timezone
  and confidence are not bound), so COALESCE preserves the stored values and
  the row never claims a timestamp it did not write.

Read side (TimelineQueryService, frontend) is untouched; timeline sections
faithfully rendered the stored value. Existing rows are repaired separately.
chris merged commit 55e546f418 into main 2026-10-04 10:31:45 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1028
No description provided.