ADR 0005: v1 Import Lifecycle and Failure Recovery
Status: proposed Date: 2026-05-10
Context
Memories imports media through browser, API, and CLI upload sessions. Original bytes must be preserved, duplicate storage must not collapse ownership, and crash recovery must never silently produce database records pointing to missing originals.
The filesystem and PostgreSQL cannot be committed atomically as one transaction, so v1 needs explicit failure-mode and reconciliation rules.
Decision
The Core API owns upload/import commit. External workers never create StoredObject, OriginalAsset, or MediaItem records.
Upload bytes first land in non-authoritative library/upload-staging. Core API then hashes and classifies each file, commits supported readable files per file, and enqueues post-commit worker jobs only after authoritative records exist.
Per-file commit is idempotent. A client file key is unique within an upload session, and commit requests use an idempotency key. Retrying upload or commit for the same file must return or resume the existing result, not create duplicate owner-owned media records.
Lifecycle
Session states:
createdreceivingcommit_requestedcommittingcompletedcompleted_with_issuesfailedcancelledexpired
File states:
receivingreceivedhashingclassifiedunsupportedcorrupt_or_unreadablecommit_pendingcommittingcommittedprocessing_queuedpreview_readyprocessing_failedfaileddiscarded
Commit Boundary
For each eligible file, Core API:
- Locks the upload file row.
- Verifies the staged file exists and matches recorded hash/size.
- Creates or reuses a final immutable stored-object path using a temporary final name and atomic rename where possible.
- Verifies final size/hash.
- In one PostgreSQL transaction, creates or reuses
StoredObject, createsOriginalAssetandMediaItem, links the upload file, records provenance, and enqueues post-commit jobs. - Deletes staging bytes asynchronously after database commit.
The preferred failure mode is an orphaned unreferenced file that can be reconciled later, not a database reference to missing original bytes.
Recovery And Cleanup
On startup or scheduled reconciliation:
- Stale
receivingfiles become failed with retry actionresend_file. received,hashing,classified, andcommit_pendingfiles with staging bytes present resume safely.committingfiles are reconciled against committed asset links and retried or failed visibly.- Leftover staging bytes for committed files are deleted.
- Orphan final files without DB references are quarantined or deleted after orphan retention by storage integrity/GC logic.
- DB stored-object records pointing to missing bytes create
storage_integrityprocessing issues. upload-stagingis ignored during fidelity restore and is never authoritative backup state.
Folder, Sidecar, Hidden, And Server-Side Import Scope
- CLI recursive upload preserves source paths as provenance only.
- Folder paths are not identity and do not create a browsable folder hierarchy in v1.
- Sidecars are explicitly ignored in v1 and should be reported in CLI dry-run.
- CLI recursive upload skips hidden files/directories and common system files by default.
- Server-side import, watch folders, and index-in-place are deferred. If later promoted, server-side import must be another upload-session client that still copies into app-managed stored objects.
Consequences
- Import is crash-recoverable without giving workers authority over original registration.
- Duplicate bytes can reuse stored objects without merging owner/provenance records.
- Cleanup and integrity scans become required operational tasks.
- Future ingestion clients reuse the same upload-session and idempotent commit semantics.
Revisit Triggers
Revisit this decision when adding resumable/chunked uploads, native mobile backup, server-side watch folders, object storage, or a separate queue/workflow system.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference