feat(api): pgvector native embeddings + HNSW kNN for person ranking (spec 043) #826

Merged
chris merged 2 commits from feature/pgvector-embedding-index into main 2026-09-22 23:30:08 +02:00
Owner

What

Native pgvector storage for face embeddings + HNSW cosine kNN (Immich-parity), with JSONB kept as source of truth.

Image (the one breaking change, handled)

  • New deploy/postgres/Dockerfile: postgis/postgis:16-3.4 (Debian) + postgresql-16-pgvector (PGDG, v0.8.6). PostGIS untouched/reverse-lookup unaffected.
  • compose builds memories-postgres from it; same PG16 + same memories-postgres-data volume → recreate keeps all data.
  • CI: postgres scope (scripts/ci/detect-scopes.sh), PR docker-build check, build-postgres + deploy-postgres jobs; deploy-api pulls the postgres image before up.
  • Rollback documented (deploy/postgres/README.md): image revert; API falls back to brute force if extension disappears.

Migration V136

CREATE EXTENSION vector; embedding_vec vector(512) trigger-maintained from the JSONB (512-length guard, NULL on invalid — write never fails); one-time backfill; HNSW index (vector_cosine_ops, m=16, ef=300); dropped the unused 568 MB GIN placeholder. No data loss possible; JSONB rows unchanged.

Consumer (first, per spec)

personRanking steps 2+3 now use EmbeddingKnnService: one SQL GROUP BY person MIN(embedding <=> q) per candidate vector — identical clamp(1 - min pairwise distance) score with ZERO vectors loaded into Java. Scope/semantics preserved (dominant model pair, active persons, accepted face annotations, null for embedding-less/zero-norm/mismatched). Merge suggestion + worker weak matching stay brute-force (next spec: worker incremental matching).

Verification

  • Parity on real pgvector (30×512-d seed): kNN top-K == brute force (ids identical, distances ≤1e-5); per-instance MIN == brute force; native ranking == brute-force ranking (incl. null-similarity case)
  • HNSW actually used: EXPLAIN Index Scan using e_hnsw, 0.48 ms @ 5k rows
  • Local image build: 794 MB, pgvector 0.8.6; PostGIS test package green on swapped image (34 tests)
  • Scoped suites: 480 tests 0 failures; people 147, places 34; 16 PG-backed test classes moved to the pgvector-capable test image (V136 runs in every Flyway chain)

Deploy note: merging this recreates memories-postgres (data preserved, verified path). Post-merge deploy run will be watched explicitly.

## What Native pgvector storage for face embeddings + HNSW cosine kNN (Immich-parity), with JSONB kept as source of truth. ## Image (the one breaking change, handled) - New `deploy/postgres/Dockerfile`: `postgis/postgis:16-3.4` (Debian) + `postgresql-16-pgvector` (PGDG, v0.8.6). PostGIS untouched/reverse-lookup unaffected. - compose builds `memories-postgres` from it; same PG16 + same `memories-postgres-data` volume → recreate keeps all data. - CI: postgres scope (scripts/ci/detect-scopes.sh), PR docker-build check, build-postgres + deploy-postgres jobs; deploy-api pulls the postgres image before up. - Rollback documented (deploy/postgres/README.md): image revert; API falls back to brute force if extension disappears. ## Migration V136 CREATE EXTENSION vector; `embedding_vec vector(512)` trigger-maintained from the JSONB (512-length guard, NULL on invalid — write never fails); one-time backfill; HNSW index (`vector_cosine_ops`, m=16, ef=300); dropped the unused 568 MB GIN placeholder. No data loss possible; JSONB rows unchanged. ## Consumer (first, per spec) personRanking steps 2+3 now use `EmbeddingKnnService`: one SQL `GROUP BY person MIN(embedding <=> q)` per candidate vector — identical `clamp(1 - min pairwise distance)` score with ZERO vectors loaded into Java. Scope/semantics preserved (dominant model pair, active persons, accepted face annotations, null for embedding-less/zero-norm/mismatched). Merge suggestion + worker weak matching stay brute-force (next spec: worker incremental matching). ## Verification - Parity on real pgvector (30×512-d seed): kNN top-K == brute force (ids identical, distances ≤1e-5); per-instance MIN == brute force; native ranking == brute-force ranking (incl. null-similarity case) - HNSW actually used: EXPLAIN `Index Scan using e_hnsw`, 0.48 ms @ 5k rows - Local image build: 794 MB, pgvector 0.8.6; PostGIS test package green on swapped image (34 tests) - Scoped suites: 480 tests 0 failures; people 147, places 34; 16 PG-backed test classes moved to the pgvector-capable test image (V136 runs in every Flyway chain) Deploy note: merging this recreates memories-postgres (data preserved, verified path). Post-merge deploy run will be watched explicitly.
docs(spec): add spec 043 artifact (pgvector embedding storage + HNSW kNN)
All checks were successful
Build and Deploy / Detect changed services and test scopes (push) Successful in 4s
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
27f40b69e7
chris merged commit a289d4a0b7 into main 2026-09-22 23:30:08 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!826
No description provided.