1 ADR 0011 immich like owner wide face recognition
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0011: Immich-Like Owner-Wide Face Recognition and People Grouping

Status: accepted Date: 2026-05-14 Notes: with generic-discovery corrective-refactor debt

Context

Memories treats people/faces as core family-memory infrastructure. The current implementation can detect faces, generate face thumbnails, store embeddings, and create/reuse candidate people, but its face_clustering job shape is still too image-local. A per-image clustering job cannot produce high-quality people groups across a household library.

Christian's product requirement is Immich-like quality for identified faces and persons: few duplicate clusters per real person, very low risk of mixing different real people into one cluster, and Super Admin controls for tuning how conservative the recognition pipeline should be.

Immich's current documented/default face recognition behavior is a useful quality reference:

  • Detection/embedding uses an InsightFace-style pipeline: RetinaFace detection, aligned crops, ArcFace embeddings, default model buffalo_l.
  • Defaults: minimum detection score 0.7, maximum recognition distance 0.5, minimum recognized faces 3.
  • Recognition is DBSCAN-derived but not plain per-image DBSCAN. It searches owner/user-scoped neighboring embeddings, assigns to an existing person if nearby faces already have a person, creates a new person only when a face is a core-like point with enough nearby faces, defers weak/non-core faces, and uses missing/nightly recognition passes to revisit faces after more detections exist.
  • Recognition waits for thumbnail/face-detection queues where practical because a larger batch of embeddings gives better initial grouping.

PhotoPrism's newer pipeline also uses explicit clustering and matching safeguards, but its approach is more cluster-table/marker matching oriented and is based on FaceNet embeddings. For Memories' family People UX, the Immich pattern is a better fit: strong ArcFace embeddings, owner-wide grouping, conservative core-point person creation, and manual owner merge/confirmation as durable truth.

This decision refines ADR 0010. ADR 0010 established that Core API owns face embeddings and candidate person matching. This ADR clarifies that high-quality grouping must be owner-wide and generation-aware, not per-photo clustering. ADR 0012 first defined collaborative GroupPerson spaces; ADR 0013 supersedes that target with shareable, forkable Person identities that can be shared with users or groups without collapsing owner-local privacy boundaries.

Options Considered

Option Summary Benefits Costs / Risks
A Keep per-image face_recognition/face_clustering worker jobs that create candidate people while processing one media item Simple job model; easy worker isolation Poor clusters; too many candidate people; cannot reliably find real people across the library; diverges from Immich quality goal
B Plain owner-wide DBSCAN batch over all embeddings, replacing candidate people each run Simple conceptual clustering; can use global context Re-runs may destabilize existing people; hard to preserve owner decisions; risk of destructive regrouping; poor incremental UX
C Immich-like owner-wide recognition: Core stores embeddings and leases owner-scoped snapshots to trusted ML workers, which return People class-instance candidates for Core validation/persistence Best alignment with quality goal; avoids noisy singleton people; preserves owner authority; keeps heavy clustering/matching off weak Core hosts; can improve worker algorithms incrementally Workers receive broad owner-scoped biometric metadata for a leased job; needs explicit generation/snapshot validation; more nuanced job orchestration
D Delegate owner-wide embedding sets to ML workers for clustering and final person assignment authority Allows Python/ML implementation flexibility; could use GPU/vector libraries Makes workers authoritative for People truth; harder backup/export/security story; risks bypassing owner confirmation/governance

Decision

Choose Option C.

Memories will model face/person recognition as an Immich-like owner-wide pipeline:

  1. Detection and embedding extraction

    • The first authoritative face ML stage is detection plus embedding extraction, not detection-only followed by separate per-image recognition.
    • ML workers may process job-scoped media and return face regions, aligned face thumbnails, and ArcFace-compatible embeddings in the same result generation.
    • Detection and embedding must use the same decoded/oriented input and aligned face crops for a media item so face boxes, thumbnails, and embeddings refer to the same detected faces.
    • Core API stores face annotations, embeddings, model profile/version, input version, and provenance as sensitive owner-scoped metadata.
  2. Owner-wide recognition / grouping

    • Person grouping is a worker-computed, Core-governed owner-scoped workflow over stored embeddings.
    • It must not be modeled as independent per-image clustering jobs that decide final people in isolation.
    • Core builds a scoped recognition snapshot from current-generation owner embeddings, active owner-local people, accepted/tentative assignments, model/profile generation, and conservative thresholds, then leases that snapshot to a trusted ML worker through the normal worker protocol.
    • The worker performs nearest-neighbor search, clustering, and matching within that snapshot and submits People class-instance candidates back to Core.
    • If nearby faces already belong to an active owner person/candidate within the configured maximum distance, the worker may propose a candidate targeting that PersonRecord.
    • If no nearby person exists, the worker may propose a new unnamed owner-private Candidate Person only when the face cluster satisfies the configured minimum recognized faces/core-point threshold.
    • Non-core/singleton/ambiguous faces should remain unassigned or deferred rather than creating noisy visible candidate people.
    • Core validates every submitted owner, face, existing-person, generation, lifecycle, and model-space reference before persisting any tentative assignment or creating any unnamed candidate person.
  3. Deferred and missing recognition

    • Recognition scheduling is Core-owned product behavior, not a timing decision the household user is expected to understand.
    • After imports/resets, Core should run owner-wide recognition automatically when enough new owner-scoped face embeddings have accumulated. The default balanced trigger is 100 new face embeddings since the last owner recognition run.
    • Core should also run a final/idle pass when the owner's face-detection/embedding backlog drains, even if fewer than 100 new embeddings were produced, so small libraries do not get stuck below the threshold.
    • User-facing settings may expose suggestion timing presets such as Fast (~25 new faces), Balanced (~100 new faces), and Conservative (~250 new faces), with copy explaining that lower values show unnamed people sooner and higher values may improve grouping quality during large imports.
    • This scheduling threshold is separate from the candidate quality threshold: by default Candidate Persons are still created only when at least 3 similar faces satisfy the configured minimum recognized faces/core-point threshold.
    • A missing-recognition pass must enqueue a new owner-scoped worker proposal job that revisits unassigned owner faces after more embeddings exist.
    • A scheduled/nightly or explicit maintenance pass may enqueue a new proposal job for unassigned faces without silently overwriting owner-confirmed people.
  4. Operator controls

    • Super Admin can tune installation-level ML defaults and future-job recognition behavior.
    • Memories defaults should match Immich's current defaults unless intentionally changed by ADR:
      • model profile: buffalo_l for face recognition when available
      • minimum detection score: 0.7
      • maximum recognition distance: 0.5
      • minimum recognized faces/core threshold: 3
    • Operator controls affect detection/recognition conservativeness and future processing. Applying new settings to existing media requires explicit reset/re-run or missing-recognition action.
  5. Owner governance remains authoritative

    • Worker-submitted People class-instance candidates and suggested assignments remain owner-private tentative unless accepted by owner action or owner-enabled auto-approval policy.
    • Super Admin tuning is not a privacy bypass and cannot accept, share, merge, or name another owner's people.
    • Owner merges, dissolves, names, removals, and accepted assignments are durable domain decisions. Recognition may suggest, but must not silently undo those decisions.
    • Workers never write PostgreSQL directly, never create final PersonRecord/PersonAssignment truth themselves, and never override owner-confirmed assignments.
  6. Generation and reset semantics

    • Face reset creates a recognition generation boundary for an owner or media item.
    • Stale jobs/results/backfills from prior generations must not recreate cleared face/person metadata.
    • Activity summaries and maintenance diagnostics should distinguish current-generation work from invalidated historical work.

The old face_recognition and face_clustering job types are compatibility/historical names only; new work must not deepen or execute their per-image semantics. The long-term job model is:

  • face_detection or a renamed equivalent performs detection plus embedding extraction for a media item.
  • owner_face_recognition or a semantically equivalent worker job performs owner-wide People grouping/matching over a Core-built scoped embedding snapshot and returns People class-instance candidates.
  • group_face_recognition or a semantically equivalent worker job may later perform group-wide linking over a Core-built snapshot of contributed, group-eligible embeddings according to ADR 0012/0013 governance.
  • face_recognition and face_clustering are not worker-executed grouping stages in fresh deployments. Historical results may remain readable for compatibility/backfill, but new Core scheduling and bundled workers must treat them as unsupported/deprecated.

The generalized discovery pipeline must not absorb People into class-occurrence rows. People belongs in the class-instance occurrence lane. face_candidate, current class_instance_candidate with domainKey=people, and historical compatibility person_assignment_candidate are People evidence adapters, not worker authority over final People truth. Inbound class-occurrence observations are therefore forbidden from carrying person IDs, face annotation IDs, embeddings, assignment IDs, or other identity-bearing fields.

As of the generic discovery corrective slices, People has a manifest-backed people class-instance domain, generic class-instance candidates, accepted generic occurrence links, owner-local generic instances, generic review UX, generic instance-management UX, and existing-instance-only auto-accept policy. Legacy person_records and person_assignments remain compatibility/read/write/sharing debt while the generic authority surface expands. This ADR's quality goal still stands: owner-wide workers compute proposals over Core-leased snapshots, while Core governs persistence, policy, and owner decisions.

During compatibility/backfill, person_assignment_candidate may carry face regions, embeddings, and proposed owner-local personRecordId links so Core can treat older worker output as class-instance occurrence candidate evidence. A submitted personRecordId is not accepted as final authority by itself: Core must validate ownership/lifecycle/generation, store the proposal as tentative unless an owner-enabled auto-approval policy applies, and preserve provenance for owner review and later reset/export. Fresh bundled workers must not produce these results.

Failed-media reprocess and new post-commit scheduling must not create new face_recognition or face_clustering work. Recovery should requeue face_detection so Core receives face regions/embeddings again, after which Core-owned scheduling can enqueue owner-scoped owner_face_recognition proposal work or explicit People maintenance can request it. Legacy job types may still be visible in old job history and compatibility result tables, but they should no longer drive current processing readiness or worker execution.

Default local worker bootstrap credentials must not advertise face_recognition or face_clustering. Existing credentials may retain those scopes for compatibility until explicitly rotated, but fresh deployments should advertise face_detection for face evidence, owner_face_recognition for owner-scoped People proposal generation, and the non-face jobs they actually execute.

Implementation Sequence

The migration should happen in foundation-first slices:

  1. Detection+embedding foundation

    • Change the ML worker face detection path to emit face boxes, detection confidence, thumbnails, embeddings, and embedding hashes for each detected face.
    • Change Core result acceptance so face_detection stores both face_annotations and face_embeddings without creating person records.
    • Keep generation/idempotency semantics so reset and stale-result guards remain effective.
  2. Disable noisy singleton candidate creation

    • Stop creating a Candidate Person merely because one face has an embedding and no current match.
    • Unknown faces remain unassigned/deferred until owner-wide recognition finds enough nearby embeddings or an owner assigns them manually.
  3. Worker owner-wide recognition proposal pass

    • Add a Core scheduling/snapshot workflow and worker handler for all and missing recognition over one owner and one face-recognition generation.
    • Use Immich-aligned settings: maximum recognition distance and minimum recognized faces/core threshold.
    • Worker proposes assignments to existing active owner people when nearby embeddings already have a person.
    • Worker proposes Candidate Persons only from sufficiently dense unknown neighborhoods.
    • Core validates and persists accepted proposals as tentative/accepted-by-policy state.
  4. Scheduling and UX

    • For small uploads, recognition may run after an idle/debounce period even if the embedding-count trigger is not reached.
    • For large imports/resets, recognition should run incrementally as new face embeddings cross the configured threshold and again when the backlog drains.
    • Add an explicit missing-recognition or “Run people grouping” maintenance action as an escape hatch, not as the primary UX.
    • Explore/Profile should show understandable states such as “detecting faces”, “grouping people”, or “people suggestions will improve as processing continues” rather than requiring users to know when clustering is safe.
  5. Legacy cleanup

    • Remove per-image face_recognition/face_clustering worker behavior from fresh bundled workers.
    • Update Admin settings, activity labels, processing issues, and worker capabilities to reflect detection/embedding and owner-wide grouping as distinct stages.

Consequences

  • People quality becomes a first-class architectural goal rather than an emergent side effect of per-image ML jobs.
  • Core API needs a recognition coordinator that can enqueue owner-scoped recognition proposal jobs, track generation/cutoff state, and apply idempotent result acceptance.
  • Face embeddings become even more central sensitive metadata. Backup/export/security documentation must continue treating them carefully.
  • Worker-side exact scans may be acceptable for early validation, but household libraries with tens/hundreds of thousands of faces will likely need an explicit local vector index or chunked worker strategy.
  • The UI should expose clear states: detecting faces, grouping people, unassigned faces, hidden/noise faces, candidate people, accepted people, and stale/reset generations where relevant.
  • Duplicate candidate people are acceptable and owner-mergeable; mixed-person clusters are the more serious failure mode and thresholds should default conservative.
  • Existing implementation tasks around face reset and face processing need refactoring: after owner-wide reset, detection/embedding should run first; recognition/grouping should run as an explicit later stage or missing-recognition pass.

Revisit Triggers

Revisit this decision if:

  • Immich materially changes its default face recognition approach or defaults.
  • Owner-wide worker recognition is too slow or the Core-built snapshot becomes too large for the v1 validation library.
  • A local vector index, chunked worker protocol, or worker-side cache introduces unacceptable operational complexity for v1.
  • A future deployment requires untrusted/third-party ML workers, changing the privacy boundary for embeddings.
  • Real-library testing shows ArcFace/buffalo_l quality is insufficient for Christian's household photos.
  • Operator tuning needs per-owner rather than installation-level defaults.