1 ADR 0005 v1 import lifecycle failure recovery
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0005: v1 Import Lifecycle and Failure Recovery

Status: proposed Date: 2026-05-10

Context

Memories imports media through browser, API, and CLI upload sessions. Original bytes must be preserved, duplicate storage must not collapse ownership, and crash recovery must never silently produce database records pointing to missing originals.

The filesystem and PostgreSQL cannot be committed atomically as one transaction, so v1 needs explicit failure-mode and reconciliation rules.

Decision

The Core API owns upload/import commit. External workers never create StoredObject, OriginalAsset, or MediaItem records.

Upload bytes first land in non-authoritative library/upload-staging. Core API then hashes and classifies each file, commits supported readable files per file, and enqueues post-commit worker jobs only after authoritative records exist.

Per-file commit is idempotent. A client file key is unique within an upload session, and commit requests use an idempotency key. Retrying upload or commit for the same file must return or resume the existing result, not create duplicate owner-owned media records.

Lifecycle

Session states:

  • created
  • receiving
  • commit_requested
  • committing
  • completed
  • completed_with_issues
  • failed
  • cancelled
  • expired

File states:

  • receiving
  • received
  • hashing
  • classified
  • unsupported
  • corrupt_or_unreadable
  • commit_pending
  • committing
  • committed
  • processing_queued
  • preview_ready
  • processing_failed
  • failed
  • discarded

Commit Boundary

For each eligible file, Core API:

  1. Locks the upload file row.
  2. Verifies the staged file exists and matches recorded hash/size.
  3. Creates or reuses a final immutable stored-object path using a temporary final name and atomic rename where possible.
  4. Verifies final size/hash.
  5. In one PostgreSQL transaction, creates or reuses StoredObject, creates OriginalAsset and MediaItem, links the upload file, records provenance, and enqueues post-commit jobs.
  6. Deletes staging bytes asynchronously after database commit.

The preferred failure mode is an orphaned unreferenced file that can be reconciled later, not a database reference to missing original bytes.

Recovery And Cleanup

On startup or scheduled reconciliation:

  • Stale receiving files become failed with retry action resend_file.
  • received, hashing, classified, and commit_pending files with staging bytes present resume safely.
  • committing files are reconciled against committed asset links and retried or failed visibly.
  • Leftover staging bytes for committed files are deleted.
  • Orphan final files without DB references are quarantined or deleted after orphan retention by storage integrity/GC logic.
  • DB stored-object records pointing to missing bytes create storage_integrity processing issues.
  • upload-staging is ignored during fidelity restore and is never authoritative backup state.

Folder, Sidecar, Hidden, And Server-Side Import Scope

  • CLI recursive upload preserves source paths as provenance only.
  • Folder paths are not identity and do not create a browsable folder hierarchy in v1.
  • Sidecars are explicitly ignored in v1 and should be reported in CLI dry-run.
  • CLI recursive upload skips hidden files/directories and common system files by default.
  • Server-side import, watch folders, and index-in-place are deferred. If later promoted, server-side import must be another upload-session client that still copies into app-managed stored objects.

Consequences

  • Import is crash-recoverable without giving workers authority over original registration.
  • Duplicate bytes can reuse stored objects without merging owner/provenance records.
  • Cleanup and integrity scans become required operational tasks.
  • Future ingestion clients reuse the same upload-session and idempotent commit semantics.

Revisit Triggers

Revisit this decision when adding resumable/chunked uploads, native mobile backup, server-side watch folders, object storage, or a separate queue/workflow system.