perf(api): replace O(n^2) map viewport clustering with spatial grid #1157

Merged
chris merged 1 commit from perf/map-viewport-api into main 2026-10-13 09:26:29 +02:00
Owner

Summary\n- GET /api/map/viewport clustered up to 5000 markers with an O(n²) BFS (~25M distance checks worst case); representativeMarker re-projected members twice.\n- Replaced with a Web-Mercator spatial grid (cell >= 96px, wrapped 3x3 neighborhood, sorted per-expansion candidates) so discovery order — member lists, centroids, representatives, cluster ids, output order — is unchanged. Contract untouched; no migration needed (EXPLAIN showed all join keys already indexed; bbox predicate is post-join COALESCE so no plain index would help).\n\n## Measured (5000 markers, testcontainer postgres)\n| | Before | After |\n|---|---|---|\n| Clustering | 133.1ms | 7.0ms (~19x) |\n| In-memory 5k | 208.4ms | 36.0ms |\n\nClustering now ~9% of a capped request (SQL dominant).\n\n## Testing\n- Equivalence matrix (zooms 0-22, fixtures 0-750, antimeridian wrap, hidden-members, representative ties) vs frozen legacy oracle\n- rtk ./gradlew :apps:api:test fully green (incl. MapQueryServiceTest 9/9)

## Summary\n- GET /api/map/viewport clustered up to 5000 markers with an O(n²) BFS (~25M distance checks worst case); representativeMarker re-projected members twice.\n- Replaced with a Web-Mercator spatial grid (cell >= 96px, wrapped 3x3 neighborhood, sorted per-expansion candidates) so discovery order — member lists, centroids, representatives, cluster ids, output order — is unchanged. Contract untouched; no migration needed (EXPLAIN showed all join keys already indexed; bbox predicate is post-join COALESCE so no plain index would help).\n\n## Measured (5000 markers, testcontainer postgres)\n| | Before | After |\n|---|---|---|\n| Clustering | 133.1ms | 7.0ms (~19x) |\n| In-memory 5k | 208.4ms | 36.0ms |\n\nClustering now ~9% of a capped request (SQL dominant).\n\n## Testing\n- Equivalence matrix (zooms 0-22, fixtures 0-750, antimeridian wrap, hidden-members, representative ties) vs frozen legacy oracle\n- `rtk ./gradlew :apps:api:test` fully green (incl. MapQueryServiceTest 9/9)
perf(api): replace O(n^2) map viewport clustering with spatial grid
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 21m58s
4b44733cd7
GET /api/map/viewport clustered up to 5000 projected markers with a BFS
whose inner loop scanned every projected marker per queue entry (~25M
distance checks worst case), and representativeMarker re-projected every
member twice. Replace the full scan with a Web-Mercator spatial grid
(cell side >= CLUSTER_DISTANCE_PIXELS = 96px via floor(mapSize/96)
columns, so any pair within 96px is always in the wrapped 3x3
neighborhood) and expand the BFS per cell; per-expansion candidates are
sorted ascending so discovery order — and therefore member lists,
centroids, representatives, cluster ids and output ordering — stays
identical to the previous algorithm. representativeMarker now projects
each member once with Stream.min-equivalent strict-less tie-breaking.

Equivalence with the pre-change algorithm is pinned by
MapClusterGridEquivalenceTest against MapClusterEquivalence, a verbatim
frozen copy of the old O(n^2) pipeline (randomized zoom/size matrices,
hidden-members mode, antimeridian wrap, representative ties), which also
benchmarks the old and new pipelines side by side.

Measured (test fixtures, JDK 21, postgres:16 testcontainer):
- 5000-marker DB-backed fixture: clustering 133.1ms -> 7.0ms (~19x);
  SQL fetch+mapping 69.3ms, so clustering drops from ~66% to ~9% of a
  request at the marker cap.
- In-memory volumes: 100: 6.8->3.3ms, 1000: 42.5->16.0ms,
  5000: 208.4->36.0ms.

No API/JSON contract change (fields, cluster id semantics, member
inclusion rules, DEFAULT_MARKER_LIMIT=5000, markerLimit semantics all
unchanged). No schema change: EXPLAIN on the production-shaped query
against live-like data shows every join key already indexed
(idx_media_items_owner_lifecycle, V157 metadata_overrides effective
fields, V040 upload_session_files, V051 derived_assets); the bbox
predicate applies to the COALESCE(location_override, mi.latitude)
expression, which no plain index can serve.

During SQL capture for the baseline the postgres log_min_duration_statement /
log_statement ALTER SYSTEM settings were briefly set on the running
memories-postgres container and then reset to defaults; no queries or
data were read or modified.
chris force-pushed perf/map-viewport-api from 4b44733cd7
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 5s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 21m58s
to ff414ce9be
All checks were successful
Wiki Sync / Sync wiki from docs (push) Has been skipped
Build and Deploy / Detect changed services and test scopes (push) Successful in 2s
Build and Deploy / Android unit tests and debug APK (push) Has been skipped
Build and Deploy / Smoke API and media worker (push) Has been skipped
Build and Deploy / Smoke web (push) Has been skipped
Build and Deploy / Smoke ML worker (push) Has been skipped
Build and Deploy / Smoke people worker (push) Has been skipped
Build and Deploy / Build postgres image (push) Has been skipped
Build and Deploy / Build plants worker image (push) Has been skipped
Build and Deploy / Build converter worker image (push) Has been skipped
Build and Deploy / Deploy postgres (push) Has been skipped
Build and Deploy / Build API image (push) Has been skipped
Build and Deploy / Build media worker image (push) Has been skipped
Build and Deploy / Build ML worker image (push) Has been skipped
Build and Deploy / Build people worker image (push) Has been skipped
Build and Deploy / Build web image (push) Has been skipped
Build and Deploy / Deploy API (push) Has been skipped
Build and Deploy / Deploy web (push) Has been skipped
Build and Deploy / Deploy media worker (push) Has been skipped
Build and Deploy / Deploy ML worker (push) Has been skipped
Build and Deploy / Deploy plants worker (push) Has been skipped
Build and Deploy / Deploy people worker (push) Has been skipped
Build and Deploy / Deploy converter worker (push) Has been skipped
Build and Deploy / Clean Docker build artifacts (push) Has been skipped
Build and Deploy / Run scoped PR tests (push) Successful in 7m15s
2026-10-13 09:15:57 +02:00
Compare
chris merged commit b1f67dec02 into main 2026-10-13 09:26:29 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
chris/memories!1157
No description provided.