1 ADR 0015 semantic search before redesigned things
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0015: Implement semantic search before redesigned Things labels

Status: accepted

Context

ADR 0014 disabled the CLIP-derived visual_concept_detection pipeline because it converted raw image/text similarity into misleading label confidence. Christian clarified the replacement priority: implement semantic search first, then redesign automatic Things labels later.

Semantic search and automatic labels are different product claims. Semantic search ranks media for a text query; it does not assert that an image definitively contains a concept. Things labels are persistent metadata and require calibrated detector/classifier confidence, provenance, and conservative UX.

Decision

  • v1 will add local semantic image embeddings for search ranking before reintroducing Things labels.
  • Core may call the local ML worker synchronously for stateless text-query embedding inference. This is a narrow exception to the worker-poll model; workers still never write authoritative database state and never own search/index truth.
  • Media image embeddings are produced asynchronously by worker jobs and accepted by Core as authoritative indexed metadata.
  • The first storage slice uses JSONB vectors plus Core-side cosine ranking for inspectability and deployment simplicity. pgvector remains the preferred future migration when scale or ranking latency requires it.
  • Search results may blend lexical filename/title/description matches with semantic similarity, but the UI must present them as search results, not detected Things.
  • Redesigned Things labels remain deferred and must be detector/classifier-backed, not CLIP similarity presented as confidence.

Consequences

  • Existing PostgreSQL Docker images can run this slice without extension changes.
  • First semantic queries may fall back to lexical results when the text embedding endpoint or model is unavailable.
  • Semantic search ranking is inspectable but not optimized for very large libraries until a vector index is introduced.
  • The ML worker gains a narrow synchronous HTTP endpoint for query embedding while retaining the async job path for media embedding generation.

Revisit triggers

  • Search latency becomes unacceptable on household-scale libraries.
  • The deployment standard includes a PostgreSQL image with pgvector available.
  • Redesigned Things labels are promoted and need their own detector-backed metadata pipeline.