1 ADR 0001 face clustering dbscan
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00

ADR 0001: Face Clustering with Custom DBSCAN

Status: Proposed
Date: 2026-07-15
Deciders: Christian Wiwie

Context

Face detection produces face embeddings — vectors in high-dimensional space. To group these into distinct persons, the system needs a clustering algorithm. The clustering behavior must satisfy three constraints:

  1. User-confirmed assignments are sacred. Once a user confirms a face belongs to a person (via the UI), that assignment must survive re-clustering indefinitely. Re-clustering must never reassign a confirmed face to a different person.
  2. Unconfirmed assignments are provisional. Faces detected and auto-assigned by the system can be reassigned when clustering re-runs (e.g., when new embeddings change cluster boundaries).
  3. Clustering must run automatically and on-demand. It should run periodically to incorporate new embeddings, and also support manual trigger from the /people page for immediate feedback.

The existing legacy pipeline used a different approach (fixed assignments only). The new pipeline must bake these constraints into the clustering layer from day one.

Options Considered

Option Summary Benefits Costs / Risks
A Custom DBSCAN with fixed-point constraints Confirmed assignments never change; unconfirmed adapt naturally; no separate "lock" tables Custom implementation effort; must handle incremental re-clustering efficiently
B Offline HDBSCAN with post-hoc overlay State-of-the-art variable density clustering HDBSCAN doesn't natively support fixed-point constraints; would need locking layer on top; more complex runtime
C Semi-supervised clustering (COP-KMeans) Native constraint support KMeans assumes spherical clusters — poor fit for face embedding space; requires k preset
D Fixed assignment only (no re-clustering) Simplest to implement Never adapts to new detections; confirmed + unconfirmed treated identically; misses grouping improvements

Decision

Option A: Custom DBSCAN with fixed-point constraints.

Custom DBSCAN variant that treats confirmed face-person assignments as immovable anchors:

  • Core algorithm: Density-based spatial clustering of applications with noise (DBSCAN). Groups face embeddings that are closely packed together (high density) into clusters. Embeddings in low-density regions are labeled as noise (unassigned).
  • Fixed-point constraint: Before clustering, confirmed embeddings are pre-assigned to their person cluster. The clustering algorithm must not move them. This is enforced by seeding cluster centers from confirmed assignments and only assigning unconfirmed embeddings to matching or new clusters.
  • Unconfirmed flexibility: Unconfirmed face embeddings are free to move between clusters or be labeled as noise when clustering re-runs.
  • Periodic execution: A scheduled task (e.g., every 5 minutes via @Scheduled) triggers re-clustering of all unconfirmed embeddings.
  • Manual trigger: The /people page exposes a "Re-cluster" button that triggers immediate clustering via the API.

This approach gives:

  • User trust: once confirmed, never undone
  • Adaptability: new detections automatically folded in on next cycle
  • Predictability: periodic + manual schedule

Consequences

  • Easier: User-facing consistency — confirmed assignments are permanent.
  • Easier: No separate permission/lock model for confirmed vs. unconfirmed at the DB level — the clustering logic enforces it.
  • Harder: Custom DBSCAN must handle incremental state — re-clustering the entire embedding set on every run must remain fast (<30s for typical library sizes).
  • Harder: Must distinguish "confirmed" vs "unconfirmed" face_occurrence rows so the clustering layer knows which are fixed.
  • Risk: If a user confirms many faces, the fixed-point set grows and eventually constrains clustering too tightly. Mitigation: user can un-confirm a face (remove fixed-point status) to let clustering re-evaluate it.

Revisit Triggers

Revisit this decision if these assumptions change:

  • Clustering runtime exceeds 30s for libraries >100k face embeddings
  • Users report that confirmed assignments prevent desired re-groupings at scale
  • A production-grade semi-supervised clustering library emerges with native DBSCAN support
  • The face embedding model changes (different dimensionality or distance metric)