test(workers): offline linkage comparison harness — production greedy clique vs constrained HAC #1033

Merged
chris merged 1 commit from experiment/linkage-comparison into main 2026-10-05 07:42:32 +02:00
Owner

What

Experiment-only module workers/people/experiments/linkage_comparison/ (+29 tests in workers/people/tests/test_linkage_comparison.py). No production code is imported or changed — no Spec-050 contract touched, no runtime behavior change.

Why

Spec 050 semisupervised-v1 groups via greedy complete-linkage cliques (_grow_greedy_clique, reusable anchors); many C-clusters end up as 2-face fragments. This harness measures whether constrained standard HAC average-linkage would group better, without changing anything.

Methods (shared outcome-gate code, exact production transcription)

Method Grouping Role
complete-clique production _semisupervised_candidates, called directly baseline
hac-average standard HAC, mean of ALL cross-pair distances ≤ threshold, group-wide confirmed-identity conflict check candidate
hac-complete standard HAC, max-linkage, otherwise identical control (isolates linkage rule from search strategy)

The clique transcription is asserted bit-identical to production (candidates + admitted set) on every sweep row.

Approved results (synthetic multimodal embeddings, 512-dim)

Pair recall @ threshold 0.45 (pair precision 100% in all listed cases):

Seed production hac-average hac-complete
20261005 36.04% 47.75% 35.14%
20261105 51.80% 74.77% 45.95%

Threshold 0.35 shows no advantage for any method (all identical at 27.93% here).

Hard-negative anchor ablation @ 0.35 (removing 0/2/4 anchor faces of the first hard-negative pair only, all other labels preserved): precision — production 83.33 / 80.95 / 76.27; hac-average and hac-complete identical at 100 / 91.40 / 84.91.

Limitations (explicit)

  • Anchor semantics differ: production reuses anchors during growth; the HAC methods partition first and check conflicts group-wide. Production-vs-HAC therefore does not isolate the linkage rule alone (hac-average vs hac-complete does). Documented confound, not resolved here.
  • Synthetic data only. No private photos, no DB, no claims about real-library behavior.
  • Pair precision 100% applies only to these synthetic cases. No universal dominance, safety, or default-threshold claim is made; the numeric threshold 0.45 is not automatically transferable.
  • Permutation sensitivity: reordering the feed can change production partitions; reviewed samples keep target/flags/scores but presentation/evidence/keys can change. No global order-invariance claimed.
  • Known cosmetic duplicate unknown counters left as-is (reviewer: non-blocking).

Verification

  • Full workers suite: 229 passed (includes 29 new experiment tests)
  • Reviewed sweep artifacts: results/sweep-20261005-005100.json / report-20261005-005100.md, experiment_code_hash a7933187dc9765f1 (verified against actual code by reviewer; Markdown regenerated identically)
  • Result artifacts stay untracked via results/.gitignore — regenerate with python3 -m experiments.linkage_comparison.run_sweep (see README; pytest env: PYTHONPATH=src)

Next step (separate work)

A design proposal for a potential average-linkage integration will be prepared as a DRAFT for Spec 050 discussion — not part of this PR, and no production change is proposed or implemented here.

## What Experiment-only module `workers/people/experiments/linkage_comparison/` (+29 tests in `workers/people/tests/test_linkage_comparison.py`). **No production code is imported or changed** — no Spec-050 contract touched, no runtime behavior change. ## Why Spec 050 `semisupervised-v1` groups via greedy complete-linkage cliques (`_grow_greedy_clique`, reusable anchors); many C-clusters end up as 2-face fragments. This harness measures whether constrained standard HAC average-linkage would group better, **without** changing anything. ## Methods (shared outcome-gate code, exact production transcription) | Method | Grouping | Role | |---|---|---| | `complete-clique` | production `_semisupervised_candidates`, called directly | baseline | | `hac-average` | standard HAC, mean of ALL cross-pair distances ≤ threshold, group-wide confirmed-identity conflict check | candidate | | `hac-complete` | standard HAC, max-linkage, otherwise identical | control (isolates linkage rule from search strategy) | The clique transcription is asserted bit-identical to production (candidates + admitted set) on every sweep row. ## Approved results (synthetic multimodal embeddings, 512-dim) Pair recall @ threshold 0.45 (pair precision 100% in all listed cases): | Seed | production | hac-average | hac-complete | |---|---|---|---| | 20261005 | 36.04% | **47.75%** | 35.14% | | 20261105 | 51.80% | **74.77%** | 45.95% | Threshold 0.35 shows **no advantage** for any method (all identical at 27.93% here). Hard-negative anchor ablation @ 0.35 (removing 0/2/4 anchor faces of the first hard-negative pair only, all other labels preserved): precision — production 83.33 / 80.95 / 76.27; hac-average and hac-complete identical at 100 / 91.40 / 84.91. ## Limitations (explicit) - **Anchor semantics differ**: production reuses anchors during growth; the HAC methods partition first and check conflicts group-wide. Production-vs-HAC therefore does **not** isolate the linkage rule alone (hac-average vs hac-complete does). Documented confound, not resolved here. - **Synthetic data only.** No private photos, no DB, no claims about real-library behavior. - Pair precision 100% applies **only to these synthetic cases**. No universal dominance, safety, or default-threshold claim is made; the numeric threshold 0.45 is **not** automatically transferable. - Permutation sensitivity: reordering the feed can change production partitions; reviewed samples keep target/flags/scores but presentation/evidence/keys can change. No global order-invariance claimed. - Known cosmetic duplicate unknown counters left as-is (reviewer: non-blocking). ## Verification - Full workers suite: **229 passed** (includes 29 new experiment tests) - Reviewed sweep artifacts: `results/sweep-20261005-005100.json` / `report-20261005-005100.md`, `experiment_code_hash a7933187dc9765f1` (verified against actual code by reviewer; Markdown regenerated identically) - Result artifacts stay untracked via `results/.gitignore` — regenerate with `python3 -m experiments.linkage_comparison.run_sweep` (see README; pytest env: `PYTHONPATH=src`) ## Next step (separate work) A design proposal for a potential average-linkage integration will be prepared as a DRAFT for Spec 050 discussion — **not** part of this PR, and no production change is proposed or implemented here.
test(workers): offline linkage comparison harness (production greedy clique vs constrained HAC)
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Run scoped PR tests (push) Has been skipped
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
625869e8cc
Experiment-only module under workers/people/experiments/linkage_comparison;
no production code path is imported or changed.

Compares the production semisupervised-v1 grouping (greedy complete-linkage
cliques with reusable anchors, exact transcription asserted bit-identical on
every sweep row) against constrained standard HAC average-linkage and a HAC
complete-linkage control, on synthetic multimodal embeddings only.

Includes: synthetic generator, three grouping backends sharing the exact
production outcome-gate code, metrics (GT-exclusive correctness, anchor
agreement, hard-negative controls, permutation sensitivity), sweep runner,
markdown report. Results (sweep/report files) stay untracked via
results/.gitignore.

229 worker tests pass (includes 29 new experiment tests).
chris merged commit bdf406263a into main 2026-10-05 07:42:32 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1033
No description provided.