ADR 0010: Core-Owned Face Embeddings and Candidate Person Matching
Status: proposed Date: 2026-05-11
Context
Superseded/refined by ADR 0011 for People grouping semantics. This ADR remains authoritative for the privacy and authority boundary: workers may produce job-scoped face embeddings, but Core API owns embedding storage, person matching/grouping, provenance, visibility, and result acceptance. ADR 0011 changes the grouping policy from immediate one-face candidate creation toward Immich-like owner-wide recognition with core-point/minimum-face thresholds.
The v1 specification requires that local face recognition/grouping assigns newly detected faces to an existing owner person when sufficiently confident, and otherwise creates or reuses an owner-private Candidate Person for unknown but similar faces.
The initial worker-oriented implementation can detect faces and submit person-assignment candidates, but it does not reliably create Candidate Persons when a newly uploaded face is dissimilar to existing known people. A tempting implementation would let the ML worker receive known people/faces and decide which person to assign. That would give workers broad access to owner-private biometric-like face data and to the person graph.
This conflicts with the Memories boundary that workers are executors and the Core API owns authoritative identity, ownership, provenance, visibility, backup/export, and deletion semantics.
Face embeddings are privacy-sensitive derived biometric metadata. They are not original media bytes, but they are also not disposable preview cache if Memories relies on them to group Candidate Persons over time.
Options Considered
| Option | Summary | Benefits | Costs / Risks |
|---|---|---|---|
| A | Worker receives known people/faces and returns final person assignments | Keeps matching work outside Core; can scale with ML workers | Broad worker access to owner face graph; harder privacy boundary; worker influences authoritative identity; more surprising backup/export semantics |
| B | Worker extracts per-face embeddings only; Core stores embeddings and performs owner-scoped matching/grouping | Preserves Core authority; workers need only job-granted media; supports Candidate Person UX without broad worker access; backup/export provenance is explicit | Core must store/query embeddings; embeddings become sensitive authoritative metadata; matching may need indexing later |
| C | Batch DBSCAN/reclustering job creates people later | Handles global clustering and potentially better grouping | New faces do not appear immediately; slow/offline clustering blocks UX; if worker-owned, still broad embedding access |
| D | Do not store embeddings; create one Candidate Person per unknown face | Simple and avoids embedding storage | Violates spec SC-040; same unknown person will fragment across uploads; poor UX |
Decision
Choose Option B for v1.
The ML worker may detect faces and compute face embeddings for the media item covered by its active job lease. The worker must not receive all existing face embeddings, all existing persons, or the owner/family person graph as a normal part of v1 matching.
The Core API stores accepted face embeddings as owner-private ML-derived metadata linked to face_annotations, with model profile/version and input-version provenance. Core API uses those embeddings for owner-scoped matching/grouping when accepting face/person candidate results or when running owner-wide recognition passes:
- Accept or create the
face_annotationfor the media item. - Persist the face embedding/fingerprint with owner, face annotation, model profile/version, input version, and provenance.
- Compare embeddings only against active embeddings/person assignments owned by the same media owner and produced by a compatible model profile/version family.
- If nearby active faces already identify an owner person/candidate within the configured threshold, create an owner-private tentative assignment to that person.
- If no nearby person exists, create a new unnamed owner-private Candidate Person only when the current recognition policy finds enough nearby owner-scoped faces to satisfy the configured core-point/minimum-face threshold.
- Otherwise leave the face unassigned/deferred for a missing-recognition pass rather than creating a noisy singleton Candidate Person.
- Similar later unknown faces may reuse the same Candidate Person through the same Core-owned matching path.
ADR 0011 defines the chosen v1 grouping policy: Immich-like owner-wide recognition using stored embeddings, conservative defaults, core-point/minimum-face thresholds, deferred non-core faces, and missing-recognition passes. If future global clustering is delegated outside Core, the privacy boundary must be revisited before allowing remote workers to receive broad owner embedding sets.
Candidate Persons remain unnamed in normal family-facing UI until the owner names them. Tentative assignments remain owner-private unless confirmed/shared according to policy.
Consequences
- New uploaded faces can appear quickly as detected face annotations/crops, while Candidate Person creation may be deferred until owner-wide recognition has enough embedding context.
- Workers remain scoped to job-granted media and do not need broad read access to existing persons/faces.
- Face embeddings become Core-managed sensitive metadata. They must be covered by backup/restore semantics and handled carefully in export documentation.
- Core API needs deterministic, testable matching thresholds and model-compatibility rules.
- PostgreSQL vector search can start simple for v1 household validation; larger libraries may require pgvector, approximate indexing, or a dedicated local index later.
- Dissolving or merging people changes person records and assignments, not the underlying face embeddings. Embeddings remain linked to face annotations for provenance and future reassignment unless the media/face metadata is deleted according to lifecycle policy.
- JSON/JSONL export should either include embeddings only in an explicitly marked sensitive section or document that embeddings are excluded from user-facing exit export while retained in fidelity database backup.
Revisit Triggers
Revisit this decision if:
- 1,000-item validation shows Core-owned matching is too slow without a vector index.
- A future deployment wants untrusted or third-party ML workers.
- Remote GPU clustering over large owner libraries becomes necessary.
- Embedding storage raises unacceptable privacy/export concerns.
- Model changes require cross-model migration, re-embedding, or compatibility mappings.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference