1 ADR 0109 api read path multi representant ranking
Forgejo Actions edited this page 2026-10-10 05:51:53 +00:00

ADR 0109: API read-path multi-representant assignment metric: persisted worker representatives replace the live single-centroid ranking

Status: accepted Date: 2026-10-10

Context

ADR 0108 switched the people worker's semisupervised-v2 identity metric to multi-representants (per-person average-linkage split at representative_split_distance=0.50, unit-mean centroid per subcluster, per-representative pessimistic target choice), but the three API read sites behind personRanking, mergeTargetRankings and mergeSuggestion kept computing the pre-0103-era single live centroid (mean of ALL the person's accepted occurrence embeddings in the shared model pair). On multi-modal persons the single centroid decays: the mean of many spread-out faces drifts away from every actual visual mode, and the ADR 0106 pessimistic margin rewards the decayed large centroid only nominally (one n_target for the whole person). Proven production consequence (2026-10, owner library): a candidate whose faces sit 0.228 cosine from target 'Rikke''s winning mode ranked 'Jutta' first at 0.362 with Rikke outside the top 6 — Rikke holds 435 accepted faces across 47 subclusters and her collapsed centroid no longer resembles any real appearance, so the UI title ranking lies against the very cards the worker mints from the same faces. Re-deriving the linkage in Java was rejected deliberately: a second implementation of the average-linkage split carries a permanent parity risk and would recompute an O(n^2)-memory decomposition per request.

Decision

PERSIST, don't recompute. The people worker already computes every person's representatives per run; it now ships them on the class_instance_candidate result envelope as an ADDITIVE top-level field personRepresentatives ([{modelName, modelVersion, personId, reps: [{repIndex, clusterSize, embedding}]}], deterministic pair-then-person order, rep_index in cluster order, full-precision embeddings) — the per-candidate envelope contract is untouched (distance stays raw, confidence == clamp(1 - round(distance, 6)), contract version unchanged, field validated fail-closed before any write and absent-legal for older workers/v1/legacy). The API persists the set into person_instance_representatives (V156: JSONB embedding source of truth plus the V136-style trigger-maintained embedding_vec pgvector mirror, unique key (owner, domain, class, model pair, person, rep_index), FK fail-closed to class_instances) FULLY REPLACED per (owner_id, domain_key, class_key) inside the SAME transaction as the candidate ingestion — an empty array is a real replacement (the run retires the stored set), an absent field leaves prior rows untouched. The three read sites score through one shared composer: per representative d_pess = d + Zsigmasqrt(1/n_source + 1/cluster_size) with the representative's OWN cluster size as n_target and the source side's bucket size as n_source, the person's key is the FIRST-minimal over its representatives (rep_index order, the worker's deterministic convention), and the reported raw distance/similarity is the pessimistically CHOSEN representative's raw distance — the exact worker choice rule (ADR 0108 B outcome, CV medoid-pess), no 'min over all reps' shortcut and no blending with the single-centroid value. For the merge surfaces (unordered instance pairs) the pessimistic minimum over the rep-ified directions applies (each direction scores one side's unit centroid against the other side's persisted representatives); personRanking is exact worker parity because its candidate side is a pending card and never carries representatives. A person WITHOUT a persisted set in the queried model pair (fresh instance, over-cap >2000-face person which the worker represents as a single centroid, owner before the first v2 run) falls back to the pre-0108 live single-centroid metric — byte-identical old behavior. candidatesPaged, countCandidates, named-target filtering (ADR 0107) and the candidate state machine are untouched; the merge-suggestion 0.35 gate keeps its formula and now reads rep distances, which shrink systematically (ADR 0108), so more pairs may pass — the operator's known post-0108 recalibration lever (suggest band ~0.45) applies here too.

Consequences

The title ranking, merge-target picker and merge suggestions agree with the worker's card metric again on multi-modal persons (the Rikke/Jutta inversion cannot recur: Rikke's winning mode scores 0.228 + a cluster-size margin instead of her decayed 0.3+ centroid). Cost: one table proportional to representatives (~97 rows for the largest 1142-face person at 0.50), one extra indexed query per request, zero per-request linkage computation; ingestion adds a delete+insert of the set inside the existing result transaction. Freshness: representatives are minutes-stale until the next completed worker run — by design and consistent with the candidate cards themselves, which carry exactly the same staleness; a person created AFTER the last run has no set and uses the live fallback until the next run (documented degradation, never an error). The API cannot represent a person whose faces changed since the last run any better than the last run's decomposition — correctness binds to worker-run freshness, not to real time. Parity is now a data contract instead of a code port: the golden test pins the API ranking to vectors produced by the production Python path (workers/people/experiments/api_ranking_golden.py), so a worker-side change to the choice rule must move the golden file consciously. The pgvector mirror exists but nothing queries embedding_vec yet — deliberately no HNSW index until a kNN read exists (the unused V101 GIN placeholder is the cautionary precedent). suggestSimilarInstances keeps the live single-centroid metric (explicit non-goal: operator dialog, not a card/picker surface). Reset flows leave the rows alone: a full clustering reset makes the NEXT run full-replace the set; until then the stale set describes the pre-reset state, matching the stale cards.