Our model router was overpaying 33x for the same results
· the plori team
TL;DR. We replaced our model router's hand-written capability scores with measured benchmark indices, and the router immediately converged on the highest-scoring model for agentic work: 47 of 47 routed decisions on a 50-task coding benchmark went to kimi-k3, at 33x the input price of the cheapest candidate. So we ran the control: the same 50 tasks pinned to that cheapest model. Routed to kimi-k3, 34 of 50 SWE-bench Verified tasks resolved; pinned to deepseek-v4-flash, also 34 of 50, at 44% of the routed run's model spend. A benchmark index gap that looks decisive on a scoreboard did not transfer to our workload. The calibration a router actually needs is outcome data from its own traffic.
Every agent on plori runs its model calls through a router: a small, fast judge model reads the shape of the request and picks a serving model from a price-ordered candidate list. The instruction to the judge is deliberately conservative. Start from the cheapest candidate. Step up only when a costlier candidate scores meaningfully higher on the dimension the task actually needs, and the task is demanding enough for the difference to matter.
Why did hand-written scores fail?
The first version of the candidate list carried hand-written capability scores, 1 to 5, written by us on vibes and partial evidence. Those scores rated one mid-price model a full tier above deepseek-v4-flash at coding. Independent measurement disagreed: on Artificial Analysis's coding index the two models were within half a point of each other, and the "weaker" one cost, at the time, about a third as much. The judge trusted our numbers and had converged on the mid-price model for essentially all coding traffic.
Hand-written scores fail in a predictable direction. They encode the reputation a model had when someone last paid attention, and reputations lag. So we deleted the prose and the 1-to-5 scores and replaced them with measured composite indices (intelligence, coding, and agentic tool use, each 0 to 100) from Artificial Analysis, snapshotted into the router's catalog with the variant matched to the reasoning effort we actually serve.
What happened when we switched to measured scores?
With measured indices in place we ran a 50-instance subset of SWE-bench Verified through the platform, router live, nothing pinned. The subset is stratified by the dataset's difficulty labels and selected by a recorded seed, so the draw is reproducible and was not chosen after seeing results.
The judge chose the same model on 47 of 47 decisions it completed: kimi-k3, the candidate with the pool's highest agentic index. Two more tasks fell back to a static default when the judge timed out, and one task never reached a model call because of a platform fault. A monoculture, again, except this time every step of it was defensible. These were real multi-step agentic coding tasks. The relevant index really was several points higher than every cheaper candidate's. The judge did exactly what we told it to do.
The catch is the price column. Public per-token prices at the time of the run:
| Candidate | Input $/M | Output $/M | Agentic index* |
|---|---|---|---|
| deepseek-v4-flash | $0.09 | $0.18 | 45.7 |
| kimi-k3 | $3.00 | $15.00 | 50.1 |
*Artificial Analysis agentic composite, 0 to 100, snapshot 2026-08-05.
A 4.4-point index gap was buying a 33x input-price step on every routed task. Whether that trade is good depends entirely on whether the gap transfers to real outcomes, and nothing in a composite index promises that it does. Benchmark composites are model-intrinsic; your workload is not.
What happened when we pinned the cheap model?
The experiment that settles it is cheap and boring: run the same 50 instances again with the model pinned to deepseek-v4-flash, router bypassed entirely. We verified from the decision log that the pinned round produced zero routing decisions. Concurrency, per-task time limit, prompts: identical. The harness itself had picked up two recovery mechanisms between the rounds; the honesty note below deals with that. A few tasks in each round (one routed, three pinned) hit platform faults unrelated to either model and were re-run cleanly; re-runs count normally in both columns.
| Basis | Routed (kimi-k3 on 47/50) | Pinned (deepseek-v4-flash) |
|---|---|---|
| Resolved, same-harness basis | 34 / 50 | 34 / 50 |
| Model spend | $39.62 | $17.30 |
| Total billed (models + compute) | $45.27 | $25.38 |
| Total tokens | 27.2M | 33.9M |
| Per-token blended cost | ~$1.46/M | ~$0.51/M |
A tie on resolve rate, 34 each way. The pinned model burned 24% more tokens to get there, which is what a weaker model grinding harder looks like, and it still finished at 44% of the model spend and 56% of the total bill.
The honesty note on the "same-harness basis" row. The pinned round ran later the same day, after a harness improvement had landed, and two new recovery mechanisms (reading a partial patch off the agent's disk after a timeout, and a short mechanical follow-up run that extracts the diff when the agent never wrote the patch file) recovered 5 additional resolved instances for the pinned round. Its headline number on the newer harness was 39 of 50. Those 5 are excluded from the comparison above because the routed round predates the mechanisms and never got the same help. Splitting the bases is the difference between a model comparison and a harness changelog.
How did the two models fail differently?
The two models did not fail the same way.
Twelve of kimi-k3's sixteen failures were wall-clock timeouts, all in one large repository whose test suite dominates the clock. Its patches, when it wrote them, were clean: 92% of the patches it produced passed the benchmark's official evaluation.
deepseek-v4-flash showed two failure modes kimi-k3 never did. Twice in 50 tasks, the upstream stream ended abnormally before a usable turn. And four times it wrote junk diffs, from just under 1MB to 13MB: the one small enough to read buried a one-file real change under 246 files of repository noise, and it failed to even apply. The other three were too large for the harness to read at all. A cheaper model fails differently. It has its own tail.
None of that tail was big enough to move the total on this subset. It might be on yours, which is rather the point of running your own control.
What a router should actually trust
The uncomfortable summary: we replaced bad hand-written numbers with good measured numbers, the router followed them faithfully, and it still made a call that a $25 experiment showed bought nothing measurable. Not because the indices are wrong. They measure what they measure, on their benchmarks, at a serving configuration that may or may not match yours. The 4.4-point agentic gap is probably real. It just did not convert into resolved tasks on this workload at this time limit.
So we are changing the routing policy in one specific way: index gaps alone will no longer justify a price step. The router's catalog will gain a first-party column, per-model resolve rates measured on our own periodic control runs, and a step up the price ladder will have to be backed by an outcome gap, not a benchmark gap. The step-up gate shipped days after this post was published; the resolve-rate column is still to come. Until it lands, the judge stays on the cheapest capable candidate unless the pair has evidence. External indices still do the job they are actually good at: catching a model that is far off the pace, and ordering candidates for tasks we have no first-party data on yet.
If you run a model router, the experiment is worth stealing. Pin the cheap model. Run your real workload twice. It costs about as much as a nice lunch, and it turns a routing debate into a table.
For the later production-router run, see the full SWE-bench Verified benchmark report: all 50 task outcomes, actual model-call distribution, cost split, wall-clock methodology, limitations, and raw JSON.
Caveats
- One round per configuration, 50 tasks each. The tie is "no measurable difference on this workload," with roughly a one-to-two task noise band, not a claim that the models are equal in general.
- One workload family (repository-scale bug fixing under a 25-minute per-task limit). Different task shapes can absolutely justify the step up; that is what the per-pair evidence requirement is for.
- The subset is 50 of the dataset's 500 instances, run on our agent stack and time limits. These are not leaderboard numbers.
FAQ
How much does a model-routing control run cost?
Ours cost $25.38 total (models plus compute) for 50 SWE-bench Verified tasks pinned to deepseek-v4-flash. The routed run through kimi-k3 cost $45.27 for the same 50 tasks. Running both rounds back to back is under $75. If you only run the pinned control (the one that answers the question), it is a single lunch-budget experiment.
Should you trust benchmark scores for model routing?
Not on their own. Benchmark composite indices measure model-intrinsic capability on a standardized evaluation, which may not match your serving configuration, your prompts, or your task distribution. In our test, a 4.4-point agentic index gap produced zero difference in resolved tasks. Use benchmark scores to screen out models that are far off the pace, and to order candidates when you have no first-party data. Use outcome data from your own workload to justify paying more.
What is the difference between a benchmark gap and an outcome gap?
A benchmark gap is the distance between two models on a third-party index (for example, 50.1 versus 45.7 on an agentic composite). An outcome gap is the distance between two models on your workload's success metric (for example, resolved tasks out of 50). The first is general and model-intrinsic; the second is specific to your tasks, prompts, and time limits. A benchmark gap can exist without an outcome gap, which is exactly what happened in our test.
How do you set up a pinned control run?
Bypass the router entirely and force every model call to the cheapest candidate. Keep everything else identical: same task set, same concurrency, same per-task time limit, same prompts. Run the workload, then compare the success metric (resolved tasks, pass rate, whatever you measure) against the routed run. Verify from your decision log that the pinned round produced zero routing decisions.
When should you use the expensive model despite a tie?
When you have per-pair outcome data showing the expensive model resolves tasks the cheap one does not, and the delta justifies the price. Different workload families (longer-horizon planning, unfamiliar codebases, multi-repo changes) may break differently. A tie on one task shape does not generalize. The point of a control run is to get the data for each pair you actually route.
Quality indices courtesy of Artificial Analysis. Tasks from SWE-bench Verified. Prices are the public per-token rates at the time of the run.