ADR 0014: Disable CLIP-Derived Visual Concept Labels Pending Semantic Search Redesign
Status: accepted Date: 2026-05-16
Context
The v1 baseline included visual_concept_detection: a local CLIP ViT-B-32__openai image embedding compared with a small local label-embedding vocabulary. The worker then mapped cosine similarity to a user-facing confidence value with (similarity + 1) / 2 and stored the top labels as visual_concept_annotations.
Validation on Christian's library showed that detected things frequently did not match the image content, and most labels clustered around approximately 0.60 confidence. Inspection showed two core problems:
- CLIP image/text similarity is a retrieval score, not a calibrated classification probability.
- The small local label vocabulary had highly overlapping embeddings, so many labels produced similar weak scores for the same image.
This conflicts with Memories' priorities of low operational surprise, inspectability, and family-friendly UX. Showing misleading “Things” with pseudo-confidence is worse than not showing the feature.
Decision
Disable the current CLIP-derived visual concept label pipeline for new work.
Specifically:
- Core API MUST NOT enqueue new
visual_concept_detectionjobs during post-commit processing or normal reprocess/reset actions. - The default Compose ML worker MUST NOT advertise the
visual_concept_detectioncapability. - Existing
visual_concept_annotationsremain historical/exportable metadata until an explicit cleanup/migration decision is made. - Worker protocol support and result acceptance may remain as compatibility code, but the current label-similarity pipeline is not a trusted v1 product capability.
Consequences
- New uploads will no longer produce misleading Things labels.
- Media detail and activity UI should treat thing detection as unavailable/disabled rather than pending.
- Existing historical labels may still appear until hidden or cleaned up by a separate explicit task.
- Reprocess/reset actions should not recreate visual concept labels.
Future Direction
Memories should separate two concepts:
- Semantic search embeddings: model-versioned image/text embeddings used for ranking freeform search results. These are not confidence-scored labels.
- Generated labels / object detections: curated candidate metadata with model-specific thresholds, provenance, acceptance/rejection state, and correction UX.
The preferred search-index direction is PostgreSQL full-text search plus pgvector first, with VectorChord or Qdrant only as measured future alternatives. Candidate semantic models include SigLIP2 for quality and UForm3 multilingual as a lightweight/local option.
Revisit Triggers
- A semantic search ADR/spec is accepted.
- A replacement local model produces calibrated or thresholdable labels on a validation set.
- The UI gains explicit candidate/provenance/rejection affordances for generated labels.
- Christian chooses to permanently purge historical visual concept annotations.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference