feat(plants): replace CLIP pre-filter with YOLO-World detector + crop pipeline #305

Merged
chris merged 3 commits from fix/plant-false-positives into main 2026-06-11 21:52:58 +02:00
Owner

Summary

Replace the CLIP-based image-level plant presence filter with YOLO-World v2 (zero-shot open-vocabulary detector) that localizes plant bounding boxes and crops them for PlantNet classification.

New Flow

  1. Image → YOLO-World (YOLOv8s-worldv2 ONNX, prompt: plant)
  2. No detections → skip (no plant)
  3. For each detection → crop bbox → PlantNet classifier
  4. Merge results from all crops

Model Choice

YOLOv8s-worldv2 ONNX (~49MB, 12.7M params) — smallest exportable YOLO-World v2 variant. Needs CLIP ViT-B/32 text embeddings (512-dim) pre-computed for plant class.

Changes

  • New: yolo_prefilter.py — YOLO-World detector with letterbox preprocessing, NMS, bbox cropping
  • Modified: main.py — replaced CLIP flow with YOLO-World crop pipeline
  • Modified: Dockerfile — changed model dir from clip/ to yolo/
  • Modified: test_clip_prefilter.py — removed CLIP integration tests (module still exists)
  • New: test_yolo_prefilter.py — 29 tests covering detection, NMS, cropping, integration

Environment Variables

  • MEMORIES_YOLO_MODEL_DIR: path to yolo/ dir with model.onnx + text_embeddings.npy
  • MEMORIES_YOLO_PLANT_THRESHOLD: confidence threshold (default 0.25)
  • MEMORIES_YOLO_IOU_THRESHOLD: NMS IoU threshold (default 0.5)

Test Results

78/78 passing (29 new YOLO tests + 49 existing).

## Summary Replace the CLIP-based image-level plant presence filter with YOLO-World v2 (zero-shot open-vocabulary detector) that localizes plant bounding boxes and crops them for PlantNet classification. ## New Flow 1. Image → YOLO-World (YOLOv8s-worldv2 ONNX, prompt: plant) 2. No detections → skip (no plant) 3. For each detection → crop bbox → PlantNet classifier 4. Merge results from all crops ## Model Choice **YOLOv8s-worldv2 ONNX** (~49MB, 12.7M params) — smallest exportable YOLO-World v2 variant. Needs CLIP ViT-B/32 text embeddings (512-dim) pre-computed for plant class. ## Changes - **New**: yolo_prefilter.py — YOLO-World detector with letterbox preprocessing, NMS, bbox cropping - **Modified**: main.py — replaced CLIP flow with YOLO-World crop pipeline - **Modified**: Dockerfile — changed model dir from clip/ to yolo/ - **Modified**: test_clip_prefilter.py — removed CLIP integration tests (module still exists) - **New**: test_yolo_prefilter.py — 29 tests covering detection, NMS, cropping, integration ## Environment Variables - MEMORIES_YOLO_MODEL_DIR: path to yolo/ dir with model.onnx + text_embeddings.npy - MEMORIES_YOLO_PLANT_THRESHOLD: confidence threshold (default 0.25) - MEMORIES_YOLO_IOU_THRESHOLD: NMS IoU threshold (default 0.5) ## Test Results 78/78 passing (29 new YOLO tests + 49 existing).
chris force-pushed fix/plant-false-positives from 7cf16b16c6
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 4s
Build and Deploy / Run full test suite (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Run main smoke checks (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
to 96a6d5744a
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 4s
Build and Deploy / Run full test suite (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Run main smoke checks (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
2026-06-11 21:51:38 +02:00
Compare
chris merged commit 7767b21a03 into main 2026-06-11 21:52:58 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!305
No description provided.