1 ADR 0016 detector backed things pipeline
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0016: Detector-backed Things pipeline

Status: accepted

Context

ADR 0014 disabled the earlier CLIP-derived visual_concept_detection pipeline because it presented raw image/text similarity as object-label confidence. ADR 0015 kept semantic embeddings for search ranking only. Christian now wants Things back, but without repeating the semantic mismatch.

Things are persistent metadata claims: “this image contains a dog/ball/car”. Search embeddings are ranking data: “this image may be relevant to this query”. Those need separate result semantics even if future implementations share low-level model artifacts.

Decision

  • Re-enable Things through a detector/classifier-backed visual_concept_detection worker job, not through CLIP label-similarity.
  • The first slice is concrete object detection only. Abstract scene/activity labels remain deferred until backed by a classifier with honest confidence semantics.
  • ML workers submit visual_concept_candidate results with conceptType, label, detector confidence, optional bounding box, and model provenance. Core remains authoritative for accepting and storing annotations.
  • The default profile is local ONNX object detection, operator-configurable, disabled only when no model is available or the operator disables it.
  • UI copy must describe detections conservatively as locally detected Things and avoid pretending the detector is infallible.

Consequences

  • semantic_embedding remains search/index data and never creates Things labels.
  • visual_concept_annotations can continue as the v1 authoritative Things table because it already stores label, confidence, bounding box, and provenance.
  • New uploads may enqueue visual_concept_detection again when a default profile exists; workers report an issue if the local model file is missing.
  • Existing historical CLIP-derived annotations may coexist but should be superseded by reset/reprocess flows before being treated as current detector-backed output.

Revisit triggers

  • We add scene/activity classifiers.
  • We need per-label confidence calibration or label ontology migration.
  • We introduce a separate object-detection table for richer detector outputs.