1 Dev Model Selection Report
Forgejo Actions edited this page 2026-09-25 16:38:07 +00:00
This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Model Selection Report

Context

Troubleshooting complex codebase issues — which LLM(s) to use for a multi-agent workflow on the Memories project.

Benchmark Data Sources

  • AA Intelligence Index (artificialanalysis.ai) — composite quality + speed + cost
  • AA Individual Evaluations — Terminal-Bench Hard, GPQA Diamond, HLE, IFBench, SciCode, etc.
  • Coding Agent Index (CA) — pass@1 across DeepSWE, Terminal-Bench v2, SWE-Atlas-QnA
  • OpenRouter Coding/Agentic/Intelligence Scores — OpenRouter's own composite
  • OpenRouter Model Page — AA benchmarks with precise decimal values
  • Cost per Intelligence Index Task — real USD cost to complete one AA evaluation task

Relevant Benchmarks for Codebase Troubleshooting

Benchmark What it measures
Terminal-Bench Hard Agentic coding & terminal use — multi-step fix tasks
Coding Agent Index (CA) pass@1 on SWE-bench-style codebase repair
Non-Hallucination Rate Avoidance of hallucinated answers when uncertain
OR-Coding OpenRouter composite coding score
Interlligence Index AA composite quality + speed + cost score

Models Compared

Available on opencode.ai Go ($10/month flat, $60 usage cap)

Modell TB-Hard Non-Hall Coding Agentic IFBench Go Requests/Mo
MiMo-V2.5 41.7% 67.8% 56.8 23.7 67.1% 150k
MiMo-V2.5-Pro 43.2% 75.5% 60.2 29.1 79.9% 16k
DeepSeek V4 Flash 35.6% 4.2% 56.2 31.1 — 158k
DeepSeek V4 Pro 46.2% 6.0% 59.4 36.4 — 17k

Key Insight

DeepSeek V4 Pro has the highest TB-Hard (46.2%) but a 6% Non-Hallucination Rate — in 94% of uncertain cases it hallucinates. This matches the observed problem of "partial fixes that are confidently wrong."

MiMo-V2.5-Pro has 3% lower TB-Hard but 75.5% Non-Hall — 12.5x better. At the same Go price, MiMo is the better choice.

Agent Model Assignment

Agent Role Model Why
Team Chef Coordinator, PO MiMo-V2.5 High volume, no coding needed, non-hall for product decisions
Scrum Master Orchestrator MiMo-V2.5 Highest volume, dispatch logic, no coding
Senior Engineer Tech Lead, Planner MiMo-V2.5-Pro Needs highest quality: plans specs and architecture for all others. IFBench +12.8% over non-Pro for instruction-following in spec writing. Non-Hall +7.7% for honest architecture reviews.
Engineer Implementer MiMo-V2.5 Solves the two main problems (TB 35.6%→41.7%, Non-Hall 4.2%→67.8%) at 9x higher volume than Pro
Test Engineer QA MiMo-V2.5 Coding 56.8 sufficient for test code review, Non-Hall 67.8% for honest edge-case identification
Designer UX/UI MiMo-V2.5 No coding needed, design knowledge independent of model tier
Architect Architecture MiMo-V2.5 Low usage, deep reasoning adequate on non-Pro

Budget Estimate (Go $60/month)

Agent Model Requests/Mo Go Cost
Team Chef MiMo-V2.5 ~5k ~$2
Scrum Master MiMo-V2.5 ~10k ~$4
Designer MiMo-V2.5 ~300 ~$0.12
Senior Engineer MiMo-V2.5-Pro ~1.5k ~$5.55
Engineer (×2) MiMo-V2.5 ~6k ~$2.40
Test Engineer MiMo-V2.5 ~1.5k ~$0.60
Architect MiMo-V2.5 ~300 ~$0.12
Total ~$15

Well within the $60/month Go cap. Leaves $45 buffer for burst usage.

Opencode.ai Go vs OpenRouter

MiMo-V2.5

Provider Input Output Cache Read
OpenRouter — DigitalOcean $0.105/M $0.28/M $0.028/M
OpenRouter — Xiaomi $0.14/M $0.28/M $0.0028/M
Opencode Go $0.14/M $0.28/M $0.0028/M

Break-Even

Scenario OpenRouter (Xiaomi) Opencode Go
10k req/mo ~$4 $10
25k req/mo ~$10 $10
150k req/mo ~$60 $10

Go is cheaper above ~25k requests/month. For the multi-agent setup (~24k/mo), Go is slightly more expensive but provides predictable flat-rate pricing and simpler billing.

History

  • 2026-07-12: Initial report. All DeepSeek V4 models evaluated from OpenRouter page.
  • 2026-07-12: MiMo-V2.5 and MiMo-V2.5-Pro added from OpenRouter page.
  • 2026-07-12: opencode.ai Go pricing and request limits added.
  • 2026-07-12: Agent model assignment for Memories multi-agent setup.