ADR 0017: Constrained Density Retargeting for People
Status: proposed Date: 2026-05-20
Context
ADR 0011 established owner-wide, DBSCAN-like face recognition/grouping: Core API stores face embeddings, assigns faces to existing owner people when nearby embeddings support that, and creates unnamed Candidate Persons only when enough nearby owner-scoped faces form a conservative unknown-person cluster. ADR 0013 established shareable, forkable Collaborative Persons, where users can contribute faces/media to a shared Person while preserving media ownership and provenance.
The first implementation of unnamed-candidate retargeting used a simpler per-face rule: for each owner-private unnamed tentative face, find the best direct pairwise match against eligible named local/shared Person evidence and retarget that face if the best distance passes a configured threshold. This is easy to explain, but it behaves differently from the normal density-based grouping that created unnamed Candidate Persons.
Christian observed this in the Morfar case: Rikke had an unnamed local Candidate Person that clearly represented Morfar. After manually moving some faces into shared Morfar, running retrospective retargeting moved only additional faces whose direct pairwise best match crossed the threshold. Many remaining faces stayed in the unnamed Candidate Person even though they were in the same local density cluster as manually moved Morfar faces. The result was technically consistent with the pairwise rule but surprising because retargeting did not behave like the clustering/grouping model.
This pushes two priorities against each other:
- Consistency / family UX: retargeting should use the same face-space structure as normal owner-wide grouping so users do not see “same unnamed cluster” and “not close enough to retarget” as contradictory behavior.
- Privacy / low surprise: automatic retargeting must not silently rewrite accepted identities, merge different real people, or expose another owner's face graph beyond explicit sharing/contribution policy.
Known algorithmic references exist for this family of behavior, especially C-DBSCAN / constrained DBSCAN and semi-supervised density clustering with must-link/cannot-link constraints. Memories should use those concepts as reference, not depend on a random implementation.
Options Considered
| Option | Summary | Benefits | Costs / Risks |
|---|---|---|---|
| A | Keep independent pairwise retargeting | Simple; already implemented; conservative per-face behavior | Inconsistent with density clustering; threshold cliffs; ignores cluster structure; surprising after manual moves |
| B | Lower pairwise retarget thresholds | Moves more borderline faces; minimal implementation change | Still not cluster-consistent; increases false-positive risk globally; threshold remains poorly calibrated as human confidence |
| C | Link a local unnamed Candidate Person to a shared Person after some faces are moved | Matches some user expectations; easy mental model | Too coarse if the local candidate contains mixed/ambiguous faces; creates a new identity-link semantic before retarget evidence is explicit |
| D | Use constrained density clustering for retargeting | Retargeting behaves like normal clustering; accepted identities are hard constraints; supports local and shared targets with provenance | More Core logic; needs careful conflict/ambiguity handling; may need review/suggestion UX for uncertain cases |
Decision
Choose Option D.
Memories will evolve unnamed-candidate retargeting from independent pairwise matching to a constrained DBSCAN-style density retargeting pass.
The retargeting algorithm MUST treat existing trusted identities as constraints while allowing only unnamed tentative faces to move:
-
Fixed anchors
- Accepted faces for active owner-local named people are fixed local identity anchors.
- Accepted contributions for eligible visible/contributable Collaborative Persons are fixed shared identity anchors.
- Fixed anchors participate in density clustering as labeled reference points, but their identity assignments are not changed by retargeting.
-
Movable faces
- Only owner-private
owner_tentativeassignments that point to unnamed active Candidate Persons are movable. - Retargeting MUST NOT move accepted assignments, named local person assignments, rejected/superseded assignments, or another owner's private assignments.
- Only owner-private
-
Cannot-link constraints
- Different fixed identity labels are hard cannot-link constraints.
- Cluster expansion MUST NOT merge components containing different fixed identities.
- If density expansion would bridge multiple fixed identities, the boundary/movable faces involved are ambiguous and MUST remain unmoved or become explicit review suggestions rather than automatic changes.
-
Density semantics
- Retargeting SHOULD reuse the same distance metric, model-space compatibility checks,
maxDistance/epsilon behavior, and minimum recognized faces/core-point semantics used by owner-wide unknown-person grouping unless an ADR explicitly changes those defaults. - This keeps “normal clustering” and “retargeting” aligned: if an unnamed group exists because of density connectivity, retargeting can use that density connectivity to attach the movable part of the group to a fixed known/shared identity when constraints allow.
- Retargeting SHOULD reuse the same distance metric, model-space compatibility checks,
-
Local vs shared effects
- If a constrained density component has exactly one eligible fixed local named identity and movable unnamed faces, Core may retarget those movable faces to that local Person as tentative/unapproved assignments according to owner policy.
- If a constrained density component has exactly one eligible fixed shared Person identity and movable unnamed faces, Core may create shared Person contributions according to the Person share/contribution policy and owner automation settings. Shared retargeting MUST NOT imply original download access or cross-audience media visibility beyond existing sharing/contribution rules.
- Shared retargeting remains separate from shared auto-approval. Retargeting may create pending/suggested contributions unless policy and owner settings explicitly allow accepted contributions.
-
Provenance and inspectability
- Retargeted assignments/contributions MUST record constrained-density provenance: algorithm name/version, model profile/version, epsilon/max distance, minimum recognized faces/core threshold, fixed anchor identities, anchor counts, source Candidate Person IDs, affected face IDs, and whether the action was automatic or review-approved.
- Ambiguous faces SHOULD be explainable in owner-facing diagnostics or future review UI.
This ADR does not require an external library. Memories should reference C-DBSCAN and mature constrained/semi-supervised density-clustering literature, but implement the small domain-specific algorithm in Core API because authorization, provenance, identity boundaries, and contribution policy are application semantics.
Consequences
- Retargeting becomes semantically consistent with owner-wide face grouping instead of being an unrelated nearest-neighbor classifier.
- The Morfar-style case becomes explainable: remaining unnamed faces may follow fixed shared Morfar anchors when density-connected and not constrained by another identity.
- Accepted/named identities become hard constraints, which protects owner decisions from automatic rewrite.
- The implementation must extract or share reusable density-clustering primitives from
OwnerFaceRecognitionServicerather than duplicating pairwise matching logic. - Retargeting results may become less conservative than pairwise thresholding. This increases the need for clear defaults, conservative cannot-link handling, and possibly review-first behavior for shared Person retargeting.
- Separate per-face retarget thresholds are no longer part of the product model. Retargeting reuses the same internal owner-wide density-clustering parameters, including epsilon/max distance and minimum recognized faces/core-neighborhood semantics, so the user-facing policy remains an enable/disable privacy and automation choice rather than an ML tuning surface.
- UI and diagnostics should eventually explain “moved by constrained density evidence”, “ambiguous because multiple fixed identities were nearby”, and “not moved because not density-connected to a fixed identity”.
Revisit Triggers
Revisit this decision if:
- Real-library testing shows constrained density retargeting mixes different real people more often than pairwise retargeting.
- The household face graph becomes large enough that exact all-pairs density expansion is too slow without pgvector or another local vector index.
- Shared Person contribution policy needs finer distinctions between suggested, pending, and accepted constrained-density retargets.
- A future model profile changes embedding calibration enough that current epsilon/min-samples defaults no longer match owner-wide grouping quality.
- Christian decides that all shared-person retargeting must be review-first rather than automatic/policy-driven.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference