ADR 0036: Owner-Local Learned Album-Shape Proposals
Status: accepted Date: 2026-05-24
Context
The first automatic-album implementation uses deterministic multi-scale-event-worthiness-v1 clustering, scoring, and filtering. That was enough to ship reviewable owner-private suggestions, but it exposes the limit of hand-authored parent/child and regional-coherence rules: real album-worthiness is owner-specific and includes the preferred size and granularity of albums, not just whether a fixed candidate ranks higher.
Christian clarified the desired long-term direction:
- the learned system should propose album-shaped media sets itself;
- it should learn from accepted/manual albums and edits how broad or granular albums should be;
- it should reduce the search space with time, but not rely on direct neighbor pairs only;
- it should not be driven by individual-photo anchor scores because albums are set-shaped memories;
- it should use people signals where helpful, but aggregate-first and privacy-scoped.
The decision affects long-term data ownership, privacy, export/restore semantics, worker boundaries, and operational complexity, so it belongs in an ADR before implementation.
Options Considered
| Option | Summary | Benefits | Costs / Risks |
|---|---|---|---|
| A. Deterministic generator plus learned reranking | Keep deterministic candidates and learn only ordering/default visibility | Smallest change; easy fallback | Preserves current heuristic shape; cannot learn album granularity or candidate boundaries well |
| B. Core-owned learned proposal | Train/score learned album-shaped windows in the Core API | Simpler deployment; no worker protocol changes | Adds ML/math complexity to Core; may not fit future model experimentation |
| C. ML-worker learned proposal, Core-governed | ML worker trains/scores owner-local proposal windows; Core owns eligibility, privacy, persistence, acceptance, and fallback | Keeps learning code in ML boundary; supports future model iteration; preserves Core authority | Requires worker protocol/API additions; operationally more complex than Core-only |
| D. Individual-photo anchor model | Rank photos first, then grow album proposals around highly scored photos | Easy to explain as “important photos” | Optimizes for interesting photos, not album-shaped memories; risks missing ordinary-but-meaningful albums |
| E. Global/cross-owner recommendation model | Learn from all owners/instances | More data; potentially stronger model | Violates local-first/privacy expectations; social graph leakage; not acceptable by default |
Decision
Choose Option C: ML-worker learned proposal, Core-governed.
The first learned implementation should use active ranking/proposal once readiness thresholds are met, not shadow-only as the target. It may still include diagnostic shadow comparison during development or rollout, but the product goal is for learned proposals to affect the default review queue after minimum feedback and model-health gates pass.
The learned proposal model should operate over overlapping time-bounded media windows as its primary search-space reduction. It should score album-shaped media sets/windows, not individual photo anchors and not fixed local_event/regional_trip_parent categories. Direct neighbor-pair boundary models may be supporting features, but they are not sufficient as the primary model.
Core remains authoritative for:
- owner-scoped eligible media extraction and authorization;
- location/person visibility filtering before features leave Core;
- suggestion persistence, state transitions, and fingerprints;
- acceptance/rejection/dismissal decisions;
- preliminary generated album creation/promotion/archive;
- export/backup semantics for durable suggestion and album state;
- deterministic fallback and active-run lifecycle.
The ML worker may own:
- training owner-local proposal models from Core-provided feature snapshots/labels;
- scoring Core-provided candidate windows;
- returning scored proposal memberships, confidence, feature contribution summaries, and model metadata.
The ML worker must never write authoritative database state directly.
First Implementation Decisions
- Rollout mode: active learned proposal/ranking after readiness thresholds; deterministic fallback before thresholds, on disablement, or on failure.
- Runtime boundary: ML worker for training/scoring; Core for governance/persistence/authorization.
- People features: include aggregate accepted-owner-local people features in the first learned proposal feature set.
- Direct identity features: do not use raw person names, raw person IDs, or person-pair flags as default model/export/explanation features. A later explicit privacy ADR is required before adding them.
- Export/restore: use minimal learned-state export. Fidelity backup preserves authoritative database state as usual, but JSON/JSONL export does not need to preserve full feature snapshots or trained model internals. Learned training state may be reset/retrained after restore/export import unless a later ADR promotes reproducible learned-state export.
Initial Feature Groups
Feature extraction should be set/window-based and owner-scoped:
- Time/window shape: span, media count, density, largest internal gap, burst count, weekday/weekend/season context, similarity to accepted album spans.
- Location shape: visible-location coverage, spread, distinct coarse buckets, return-to-place pattern, multi-scale nearby familiarity, similarity to accepted/rejected album location profiles, no inferred
home/worklabels by default. - People aggregate features: accepted people count, media-with-accepted-people ratio, unknown-face ratio, people continuity, people composition novelty, rare accepted person count/presence, accepted/rejected album people-profile similarity.
- Media/noise features: photo/video mix, low-signal or screenshot ratio where known, burst/duplicate density, representative-media availability.
- Manual album and decision history: overlap with accepted/manual albums, similarity to rejected/dismissed proposals, edit distance from accepted-after-edit suggestions, split/merge/granularity signals.
Label Semantics
Feedback labels are not only binary accept/reject labels:
- accepted suggestion or comparable manual album: positive proposal example;
- rejected suggestion: strong negative for a similar proposal;
- dismissed suggestion: weak negative unless repeated;
- edit-before-accept or post-accept media removal/addition: membership/window-boundary feedback;
- manual split: signal that similar future windows may be too broad;
- manual merge or broad manual album: signal that similar future windows may be too narrow;
- covered-by-existing-album/manual overlap: ambiguous; may mean “already organized,” not “bad proposal.”
Export / Backup Consequences
Fidelity backup remains PostgreSQL plus stored objects/config/version metadata and therefore preserves whatever learned tables exist in the database.
JSON/JSONL export should remain an inspectable exit artifact focused on durable user/domain state. For the first learned proposal implementation, export should include:
- accepted/rejected/dismissed suggestion decisions;
- accepted album links and provenance;
- proposal/model profile identifiers where they affected durable suggestions;
- coarse score/provenance shown to the owner.
It does not need to include full feature snapshots, model weights, or enough training internals to reproduce learned behavior exactly after export/import. A restored/imported system may retrain from preserved durable user decisions where possible.
Consequences
Positive
- Lets Memories learn owner-specific album granularity instead of hardcoding parent/child semantics.
- Keeps Core authoritative for privacy, persistence, and album mutation rules.
- Moves model experimentation into the ML worker boundary where it belongs.
- Allows people signals to improve family-memory suggestions without exposing raw person identity by default.
- Keeps JSON export simpler and less privacy-sensitive for the first learned slice.
Negative / Costs
- Requires ML worker protocol/API additions for feature batches, labels, training, scoring, and model health.
- Active learned proposal can affect UX sooner, so readiness thresholds and fallback behavior must be robust.
- Minimal export means learned behavior is not exactly reproducible from JSON export alone.
- Aggregate people features still encode social patterns and require careful logging/export controls.
- Time-window search can be expensive for large owner libraries and needs explicit caps/indexing.
Privacy / Security Impact
- Training/scoring inputs are sensitive movement and social-graph metadata. Core must filter them by owner authorization before sending to the ML worker.
- Worker credentials and job leases must be scoped; the worker must not gain broad database access.
- Logs must avoid raw coordinates, person names, raw person IDs, and detailed feature vectors unless explicit debug mode is enabled.
- Cross-owner, household-wide, or global learned proposal sharing is rejected by default and requires a future ADR plus explicit opt-in.
Revisit Triggers
Revisit this decision if:
- per-owner feedback is too sparse for active learned proposal to outperform deterministic fallback;
- active learned proposal repeatedly hides meaningful albums or creates noisy proposals;
- ML worker operational complexity becomes too high for self-hosted deployments;
- export/import users need reproducible learned-state transfer rather than reset/retrain behavior;
- direct person identity features become necessary enough to justify a dedicated privacy ADR;
- candidate-window scoring cannot scale to 50k+ owner media without a different search/indexing strategy.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference