1 ADR 0020 core owned typed discovery domains
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0020: Core-Owned Typed Discovery Domains

Status: accepted Date: 2026-05-22

Context

The pre-1.0 baseline includes durable seams for local ML and worker-based processing:

  • Core API owns authorization, media access grants, job leasing, result acceptance, sharing visibility, backup/restore, and export semantics.
  • Workers execute jobs and submit candidate results; they never write authoritative database rows directly.
  • MLModelProfile records preserve model/runtime provenance.
  • ADR 0009 rejects a generic plugin system while preserving explicit modular boundaries.
  • ADR 0016 allows detector/classifier-backed Things but rejects CLIP-derived label similarity as persistent metadata.

Post-v1 visual discovery ideas such as birds in photos, plant species, OCR, landmarks, pets, documents, and richer Things all need some shared architecture. They also carry privacy and operability risks: discovered labels/text/species can reveal sensitive context, model execution can be expensive, and cloud-backed services can leak media or metadata outside the home. The target architecture must also keep Core API thin in computation: replaceable workers perform expensive discovery work, while Core validates, governs, persists, and exposes decisions.

The durable product concept is not a model architecture, weight file, runtime, or preprocessing pipeline. Those will change quickly. The durable concept Memories needs to own is an observation: a producer claims that a media item, region, frame range, or segment contains something from a known taxonomy or class namespace, with confidence, scope, provenance, and visibility policy.

This also exposes an important distinction:

  • Class occurrence: an observation that a media scope contains a class, taxon, text span, object category, or similar concept, for example Bellis perennis, robin, car, or passport.
  • Class-instance occurrence: an observation or candidate link to a durable real-world instance of a class, for example William, Holger, our Opel Astra, or Fido.

Class-instance identities need ownership, lifecycle, sharing, merge/split, representative media, aliases, contribution policy, and stronger governance. People are the first high-value class-instance domain; future domains such as household pets or an owned car may need similar semantics. Generic discovery must not flatten those identities into ordinary taxonomy labels, but People must also not remain a permanent bespoke exception outside the generic class-instance lane. ADR 0037 extends this target to collaborative/shared class instances so People collaboration does not become a permanent one-off subsystem.

The architecture tension is:

  • Flexibility: future domains should not require a new one-off subsystem for every detector/classifier.
  • Control: Memories must not become a plugin host where arbitrary code defines authoritative metadata, bypasses sharing policy, or weakens backup/export inspectability.

Options Considered

Option Summary Benefits Costs / Risks
A. Generic plugin framework Allow third-party plugins to define jobs, schemas, UI, and metadata behavior Maximum extensibility; external ecosystem possible Violates ADR 0009; requires sandboxing, compatibility policy, secrets/filesystem controls, support burden, and strong trust model before core product needs it
B. One-off domain implementations Build Birds, Plants, OCR, etc. as separate bespoke subsystems Simple for first narrow domain; no abstraction upfront Duplicates enablement, policy, provenance, export, reset, visibility, and Explore/search semantics; migration cost grows with each domain
C. Reuse visual_concept_annotations for all labels Store all discoveries as concept labels with confidence and optional bounding box Fastest path for visual labels; reuses v1 table Too weak for OCR text, taxonomy candidate lists, multi-stage crop/classification provenance, domain-specific privacy defaults, corrections, and schema migration
D. Core-owned typed observation domains Core registers known observation schemas and policies; workers or external/local producers submit typed candidate observations; Core validates/accepts results Balances extensibility with authority; preserves privacy/backup/worker boundaries; avoids generic plugins and model-runtime lock-in Requires new observation/result model and schema discipline; less open-ended than plugins; may be over-engineering if only one domain is ever built
E. Manifest-driven class occurrence and class-instance lanes Workers advertise versioned manifests for both lanes; Core auto-registers valid disabled domains, owns policy/thresholds/final decisions, and renders generic UX Preserves thin Core and dynamic domain addition without per-domain Core code; supports People/pets/vehicles as generic instance domains Requires manifest registry, standard result models, generic instance storage/review, and stricter UX contracts

Decision

Adopt manifest-driven class occurrence and class-instance discovery lanes as the target architecture for generalized discovery.

Core should store and govern class occurrences and class-instance occurrence candidates/instances. It should not deeply model every producer's architecture, weights, preprocessing, or runtime unless Core itself schedules that producer through the existing worker system and needs enough profile metadata for provenance and operations. Workers perform discovery compute; Core validates standard submissions, applies Admin/owner policy, persists authoritative state, and provides generic UX.

This is the accepted target architecture and has been implemented in narrow additive slices. It is still not approval to implement a broad plugin framework. Birds and Plants remain research probes until one is explicitly promoted by a feature spec; plants_image is currently an inbound-only contract for operator-provided local producers, not a bundled runtime/model decision.

Rules

  1. No generic plugin system: Discovery domains are not arbitrary plugins. A domain must have a Core-validated manifest with key, lane, standard result model, class set behavior, confidence semantics, privacy defaults, export shape, and migration story before implementation.
  2. Observation boundary: Core's generic responsibility is accepting, validating, storing, governing, exposing, and exporting typed observations. Model architecture, weight packaging, preprocessing, and runtime details belong to the producing worker/tool/provider except where needed as provenance.
  3. Core authority remains non-pluggable: Core API owns domain registration/configuration, candidate validation, accepted result state, reset/reprocess, sharing visibility, search/Explore authorization, backup/restore, export, and processing issues. When Core schedules local work, it also owns job scheduling and media access grants.
  4. Workers/producers remain executors: Workers or external/local tools may submit candidate observations, but they must not write authoritative discovery rows directly, mutate originals, enumerate media outside their authorization, or bypass media grants/API authorization.
  5. No per-domain Core services: Core may gain reusable platform features, validators, policies, migrations, and UX primitives, but must not add one-off application code for each domain. Domain-specific Core code is temporary architecture debt unless explicitly approved with a corrective trigger.
  6. Manifest registration: Trusted workers/providers may advertise valid manifests. Core stores normalized manifest snapshots, auto-registers valid new domains or versions as Admin-visible and disabled/available, and never auto-activates domains or manifest versions. Super Admin owns enablement and activation.
  7. Persistent claims require domain semantics: Semantic embedding similarity alone must not create persistent species/object/text/domain labels. Accepted discovery claims require a model/process with documented confidence meaning and provenance.
  8. Class occurrence and class-instance lanes are separate but both generic: Class occurrence results represent observations of classes. Class-instance domains such as People, pets, and owned vehicles use generic instance primitives for candidates, instances, owner decisions, aliases, merge/split, lifecycle, visibility, and auto-accept policy. Collaborative/shared class instances are also a generic target concept per ADR 0037, not a People-only exception.
  9. Class set and class key rules: Domains declare open or closed class sets. Closed class sets include allowed classes. Class-instance candidates must declare exactly one valid governance class key. Core does not model a universal class hierarchy in this ADR.
  10. Instance labels are Core/user-owned: Workers use opaque candidate keys and may submit supporting class/subclass evidence and occurrence references. Worker-provided labels must not become authoritative household-facing instance names by default.
  11. Core-owned confidence and policy: Core defines confidence semantics and Admin-controlled thresholds/auto-accept policies. Workers emit normalized confidence and evidence; they do not decide accepted/tentative/rejected truth.
  12. Super Admin configures; owners decide: Super Admin owns technical settings, producer trust, sensitivity, resource limits, and thresholds. Final user-governed decisions belong to media owners or relevant instance owners. Admin is not a privacy-bypass reviewer for owner-private results.
  13. Opt-in and conservative visibility: Domains are disabled unless explicitly enabled by Super Admin. Sensitivity policies may require owner opt-in/opt-out or review before processing, surfacing, search indexing, sharing, or auto-accept. Sensitive domains, especially OCR/document-like and identity domains, default conservatively.
  14. Local-first default: Core/default discovery should work with local producers. Cloud/API-backed providers, if ever supported, must be explicit opt-in provider profiles with privacy disclosure and must not be required for core functionality.
  15. Versioned provenance: Results must record domain key, manifest/schema version, taxonomy/class namespace and version where applicable, input media version, producer provenance, confidence semantics, scope/region/segment if applicable, and accepted/superseded/corrected state.
  16. Inference strategy is producer provenance, not Core truth: Image discovery domains must support whole-image and region-scoped observations. If Core schedules local workers, the selected profile may declare whether inference uses whole-image classification, region detection, crop/region classification, or a detector-then-classifier chain, but accepted Core metadata remains the observation and its provenance.
  17. Evidence-first generic UX: Enabled domains must be renderable without domain-specific frontend code in media detail, overlay markers, Explore/Search, result media walls, review queues, and Admin surfaces. Class-instance review must show representative thumbnails/crops/occurrence context, not only labels or opaque IDs.
  18. Multi-stage provenance: When a detector-then-classifier chain is used, accepted results must preserve enough provenance to identify both stages and their model/profile versions if that information is available from the producer.
  19. Inspectable migration/export: Domain manifests, configuration, accepted results, candidates, and instance state must be preserved by fidelity backup and represented in JSON/JSONL export. Model binaries remain cache/runtime artifacts unless a later ADR marks a class as required restore state.

Implemented Narrow Slices

The current platform implementation includes:

  • versioned discovery manifest registration, storage, and explicit Super Admin activation;
  • class-occurrence discovery_results with Core-owned confidence/policy/visibility and reset/reprocess behavior;
  • authorized inbound class-occurrence observations for enabled domains;
  • generic media-detail overlay markers with domain/lane/state filters;
  • generic class-instance candidates, accepted occurrence links, review decision history, owner-private instances, display names/aliases, archive/restore, same-owner merge/split, and existing-instance-only auto-accept policy;
  • generic owner-facing review and instance-management web surfaces.

This implementation is intentionally platform-shaped, not domain-specific. It should be extended by adding valid manifests, producer profiles, and generic UX behaviors, not by adding one-off Core services for each new domain.

Remaining Corrective Debt

People is the first class-instance domain and is partially bridged into this architecture. Legacy People tables and some People-specific read/write/curation paths remain compatibility debt while generic authority expands. That debt must not be used as a precedent for new non-People class-instance domains. The domain-generic collaborative/shared class-instance model from ADR 0037 also remains future work; current People collaboration is a transitional source and bridge, not the permanent generic design.

Consequences

Positive

  • Memories can add discovery domains without weakening v1 ownership, sharing, backup, or worker boundaries.
  • Future Birds/Plants/OCR work has a place to attach schemas, taxonomy/class namespaces, confidence semantics, producer provenance, and privacy policy without committing Core to current model runtimes.
  • Birds/Plants can evaluate existing models on their own terms without prematurely choosing whole-image versus crop-based inference as a permanent domain constraint.
  • People, pets, vehicles, and future household identities share one target class-instance architecture rather than becoming separate Core subsystems.
  • Search and Explore can consume Core-authorized summaries rather than hidden client-side metadata.
  • Model upgrades and taxonomy changes can be handled through explicit reprocess/supersession instead of silent metadata rewrites.
  • The architecture stays compatible with ADR 0009's "modular seams, not plugins" stance.

Negative / Costs

  • A generalized DiscoveryDomain/DiscoveryResult/class-instance model adds complexity before a first full non-People identity domain is proven.
  • Supporting multiple image inference strategies adds schema and provenance complexity even before the first model is chosen.
  • Strong schema discipline is required; otherwise typed JSON payloads can become an uninspectable dumping ground.
  • The generic class-instance model is substantially larger than the current People-specific tables and will require careful migration.
  • Current People-specific owner_face_recognition and legacy People read/write surfaces are architecture debt and must continue converging onto generic class-instance authority.
  • Keeping v1 Things in visual_concept_annotations while adding discovery_results may temporarily duplicate concepts.

Operational Impact

  • Admin/operator UI will need domain availability, model installation, enabled/paused/disabled state, job counts, concurrency/resource controls, and privacy/license disclosure.
  • Discovery reprocessing must be resumable, throttleable, and visible in processing/operator status.
  • Manifest updates must be visible and require Admin activation; worker updates alone must not change active schemas or UX.
  • Domains with external providers require explicit disclosure of data leaving the instance and should be off by default.

Security And Privacy Impact

  • Domain result visibility must be checked by Core before exposing labels, counts, facets, cards, search hits, or autocomplete values.
  • Sensitive domains can leak private context even when the media itself is shared; therefore domain sharing policy must be separate and conservative.
  • Location/date priors used by species classifiers must be policy-controlled and recorded as provenance.
  • Worker-generated instance labels are not trusted identity truth; users need scoped media evidence before naming/merging/splitting candidates.

Revisit Triggers

Revisit this decision if:

  • The first implemented non-v1 discovery domain cannot be expressed without excessive domain-specific exceptions.
  • A credible need for third-party domains emerges and Memories has a mature sandboxing/compatibility/security story.
  • Worker protocol changes become incompatible with worker-protocol-v1 additive evolution.
  • Export/restore requirements require model binaries or taxonomy datasets to become authoritative restore artifacts.
  • Cloud-backed inference becomes a desired first-class feature despite the current local-first privacy posture.
  • Existing v1 visual_concept_annotations need migration into the generalized model.
  • A future class-instance domain such as pets or owned vehicles cannot reuse the People-style identity boundary without excessive special cases.
  • The current People-specific owner-face-recognition path is not refactored into the generic class-instance model before additional instance domains are started.