Table of contents
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
Model Selection Report
Context
Troubleshooting complex codebase issues — which LLM(s) to use for a multi-agent workflow on the Memories project.
Benchmark Data Sources
- AA Intelligence Index (artificialanalysis.ai) — composite quality + speed + cost
- AA Individual Evaluations — Terminal-Bench Hard, GPQA Diamond, HLE, IFBench, SciCode, etc.
- Coding Agent Index (CA) — pass@1 across DeepSWE, Terminal-Bench v2, SWE-Atlas-QnA
- OpenRouter Coding/Agentic/Intelligence Scores — OpenRouter's own composite
- OpenRouter Model Page — AA benchmarks with precise decimal values
- Cost per Intelligence Index Task — real USD cost to complete one AA evaluation task
Relevant Benchmarks for Codebase Troubleshooting
| Benchmark | What it measures |
|---|---|
| Terminal-Bench Hard | Agentic coding & terminal use — multi-step fix tasks |
| Coding Agent Index (CA) | pass@1 on SWE-bench-style codebase repair |
| Non-Hallucination Rate | Avoidance of hallucinated answers when uncertain |
| OR-Coding | OpenRouter composite coding score |
| Interlligence Index | AA composite quality + speed + cost score |
Models Compared
Available on opencode.ai Go ($10/month flat, $60 usage cap)
| Modell | TB-Hard | Non-Hall | Coding | Agentic | IFBench | Go Requests/Mo |
|---|---|---|---|---|---|---|
| MiMo-V2.5 | 41.7% | 67.8% | 56.8 | 23.7 | 67.1% | 150k |
| MiMo-V2.5-Pro | 43.2% | 75.5% | 60.2 | 29.1 | 79.9% | 16k |
| DeepSeek V4 Flash | 35.6% | 4.2% | 56.2 | 31.1 | — | 158k |
| DeepSeek V4 Pro | 46.2% | 6.0% | 59.4 | 36.4 | — | 17k |
Key Insight
DeepSeek V4 Pro has the highest TB-Hard (46.2%) but a 6% Non-Hallucination Rate — in 94% of uncertain cases it hallucinates. This matches the observed problem of "partial fixes that are confidently wrong."
MiMo-V2.5-Pro has 3% lower TB-Hard but 75.5% Non-Hall — 12.5x better. At the same Go price, MiMo is the better choice.
Agent Model Assignment
| Agent | Role | Model | Why |
|---|---|---|---|
| Team Chef | Coordinator, PO | MiMo-V2.5 | High volume, no coding needed, non-hall for product decisions |
| Scrum Master | Orchestrator | MiMo-V2.5 | Highest volume, dispatch logic, no coding |
| Senior Engineer | Tech Lead, Planner | MiMo-V2.5-Pro | Needs highest quality: plans specs and architecture for all others. IFBench +12.8% over non-Pro for instruction-following in spec writing. Non-Hall +7.7% for honest architecture reviews. |
| Engineer | Implementer | MiMo-V2.5 | Solves the two main problems (TB 35.6%→41.7%, Non-Hall 4.2%→67.8%) at 9x higher volume than Pro |
| Test Engineer | QA | MiMo-V2.5 | Coding 56.8 sufficient for test code review, Non-Hall 67.8% for honest edge-case identification |
| Designer | UX/UI | MiMo-V2.5 | No coding needed, design knowledge independent of model tier |
| Architect | Architecture | MiMo-V2.5 | Low usage, deep reasoning adequate on non-Pro |
Budget Estimate (Go $60/month)
| Agent | Model | Requests/Mo | Go Cost |
|---|---|---|---|
| Team Chef | MiMo-V2.5 | ~5k | ~$2 |
| Scrum Master | MiMo-V2.5 | ~10k | ~$4 |
| Designer | MiMo-V2.5 | ~300 | ~$0.12 |
| Senior Engineer | MiMo-V2.5-Pro | ~1.5k | ~$5.55 |
| Engineer (×2) | MiMo-V2.5 | ~6k | ~$2.40 |
| Test Engineer | MiMo-V2.5 | ~1.5k | ~$0.60 |
| Architect | MiMo-V2.5 | ~300 | ~$0.12 |
| Total | ~$15 |
Well within the $60/month Go cap. Leaves $45 buffer for burst usage.
Opencode.ai Go vs OpenRouter
MiMo-V2.5
| Provider | Input | Output | Cache Read |
|---|---|---|---|
| OpenRouter — DigitalOcean | $0.105/M | $0.28/M | $0.028/M |
| OpenRouter — Xiaomi | $0.14/M | $0.28/M | $0.0028/M |
| Opencode Go | $0.14/M | $0.28/M | $0.0028/M |
Break-Even
| Scenario | OpenRouter (Xiaomi) | Opencode Go |
|---|---|---|
| 10k req/mo | ~$4 | $10 |
| 25k req/mo | ~$10 | $10 |
| 150k req/mo | ~$60 | $10 |
Go is cheaper above ~25k requests/month. For the multi-agent setup (~24k/mo), Go is slightly more expensive but provides predictable flat-rate pricing and simpler billing.
History
- 2026-07-12: Initial report. All DeepSeek V4 models evaluated from OpenRouter page.
- 2026-07-12: MiMo-V2.5 and MiMo-V2.5-Pro added from OpenRouter page.
- 2026-07-12: opencode.ai Go pricing and request limits added.
- 2026-07-12: Agent model assignment for Memories multi-agent setup.
Memories Wiki
Using Memories
- Accounts, the Library Hub, and Data Export
- Albums and Automatic Suggestions
- Explore and Discovery Domains
- The Media Viewer
- People and Faces
- Places and Map
- Plants (In Development)
- Datenschutzerklärung (Privacy Policy)
- Fast, Progressive Image Loading
- Search
- Sharing and Households
- Timeline and Photos
- Uploading and Import
Self-Hosting
- Backup and Restore
- CLI Password Reset
- Deployment Architecture
- Known Issues and Operator Diagnostics
- Map Tiles and Place Data Privacy
- Storage Layout
- Upgrades and Versioning
- Workers
Reference