1 ADR 0006 v1 storage backends layout integrity gc
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0006: v1 Storage Backends, Layout, Integrity, and Garbage Collection

Status: proposed Date: 2026-05-10

Context

Memories must preserve original media bytes while staying operable for self-hosters. Users may place the library on SSD, HDD, CIFS, or NFS, and future deployments may want S3/object storage. At the same time, v1 should avoid turning storage abstraction into broad operational complexity before the local filesystem path is correct.

Decision

v1 models storage through a StorageBackend boundary but implements only the filesystem backend.

SSD, HDD, CIFS, and NFS are supported as filesystem deployments when the operator provides required semantics: read-after-write consistency, atomic rename within a directory, stable metadata/locking behavior, and durable writes.

S3/object storage is modeled as a future backend behind backend-neutral storage keys, not a v1 implementation requirement.

Stored Object Layout

SHA-256 is the v1 canonical content hash.

Filesystem stored object keys use:

stored-objects/sha256/<first-2>/<next-2>/<sha256>.<normalized-ext>

Example:

stored-objects/sha256/ab/cd/abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.jpg

Rules:

  • contentHashAlgorithm = sha256.
  • contentHash is lowercase 64-character hex.
  • Extension is operator/tooling friendly, not identity.
  • Dedup key is (storageBackendId, contentHashAlgorithm, contentHash, byteSize).
  • Original filename and source path remain owner/provenance metadata, not storage identity.

Write Rules

Core API writes upload bytes into upload-staging first. During import commit, Core API writes to a temporary file in the target final directory, verifies size and SHA-256, atomically renames to the final key where supported, fsyncs where practical, and only then records authoritative database state.

Existing final stored object keys are never overwritten. Final stored object bytes are never mutated.

The accepted failure mode is an orphaned unreferenced file. A database reference to missing original bytes is a preservation issue.

Integrity

The system supports integrity checks for:

  • existence;
  • byte size;
  • SHA-256 hash;
  • invalid stored object paths/keys;
  • orphan files;
  • quarantined files.

Preservation-impacting issues create visible ProcessingIssue records with storage_integrity class.

Garbage Collection

StoredObject bytes can be physically deleted only when no active, trashed/restorable, pending-delete, or active-worker references remain. GC marks objects gc_pending, waits a grace period, then performs a final reference recheck before unlinking.

Orphan final files are quarantined or retained for operator inspection before deletion. v1 does not auto-adopt orphan files into the database.

Derived Assets

Derived assets are cache and use generation-versioned keys:

derived/v1/<derived-type>/<media-id-prefix>/<media-id>/<generation-version>/<variant>.<ext>

Derived records include source stored object/hash snapshots so they can be marked stale and regenerated when generation rules change.

Workers do not write final derived paths directly in v1. Binary worker outputs are uploaded as Core API job-scoped artifacts; Core API validates and promotes accepted artifacts into final derived/v1/... keys.

Consequences

  • v1 remains boring and inspectable for self-hosters.
  • Future S3/object storage can be added without changing domain identity.
  • CIFS/NFS support is possible but documented as dependent on filesystem semantics, not guaranteed for every mount configuration.
  • Integrity and GC become explicit operational workflows.

Revisit Triggers

Revisit this decision when adding object storage, multi-library storage tiers, remote workers writing artifacts directly, cloud backup targets, or content-hash migration.