Stay connected with KayaToday, follow us on Instagram and Facebook for the latest news and reviews delivered straight to you.
When Alibaba launched Qwen 3.8-Max this week, it arrived with a marketing claim that positioned the model as second only to Claude Opus 5 in coding-agent performance. Look at the fine print on Alibaba’s own launch-day table, though, and the picture is already murkier: the model leads on just one of 12 coding-agent rows. Then look at an independent benchmark harness called VulcanBench, which apparently tested the Preview version, and the story flips almost entirely. Qwen 3.8-Max’s best-effort setting lands mid-pack. Its default setting comes last.
Both results are defensible. Neither is fabricated. The gap between them comes down to a single variable that almost no headline benchmark reports clearly: how much time and how many tokens the model was allowed to spend on each task. Alibaba’s numbers carry a five-hour timeout on coding tasks and up to 12 hours per run on PaperBench. VulcanBench allowed between 45 and 60 minutes of wall-clock time. A time budget between five and 16 times larger explains the enormous difference in outcomes. That asymmetry is not a scandal, but it is a serious problem for anyone trying to choose a model for production use.
Price per token has stopped predicting what you will actually pay
The first comparison most outlets ran on Qwen 3.8-Max was a price-per-token table, because that was the only data available in the first few days. The numbers are not flattering. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, lists at 14 cents per million input tokens and 28 cents per million output tokens. Qwen 3.8-Max lists at $2 input and $6 output. Kimi K3 sits at $3 and $15.
Those figures used to be a reasonable proxy for cost. They no longer are, for a reason specific to reasoning models. Reaching a result requires thinking tokens, and a model that spends most of its token allowance on internal reasoning can hit a token cap before it writes any answer at all. The output you receive is empty, indistinguishable from a complete failure, and you have paid for a full run anyway.
Artificial Analysis has published the clearest measurement of how this compounds in practice. Running its Intelligence Index on DeepSeek-V4-Flash at maximum effort consumed 210 million output tokens against a class median of 100 million. Absolute cost stayed manageable because the per-token price was so low. But verbosity costs time as well as money, and depending on your service-level requirements, latency alone can make a model unusable regardless of its price card.
Most of your failures are a configuration problem, not a model quality problem
There is a critical distinction that almost no benchmark harness makes and almost no leaderboard reports: the difference between a run that produces a wrong answer and a run that simply runs out of budget. They look identical in aggregate pass-rate statistics, but they have entirely different fixes. A wrong answer might call for a better model or a better prompt. A budget exhaustion calls for a larger token allowance, a tighter reasoning-effort setting, or a different task decomposition strategy.
Long-Horizon-Terminal-Bench, published in July, ran 17 frontier models across 46 tasks through a shared harness with one 90-minute attempt each. Timeouts accounted for 79% of unresolved runs. Agents that stopped on their own accounted for 19%, and harness errors for 3%. The authors are careful to note that the timed-out runs were not close to finishing, with mean reward between 0.10 and 0.35, so simply giving models more time would not automatically produce success. The lesson is more fundamental: benchmarks are implicitly measuring time efficiency whether or not they say so, and a leaderboard that does not disclose its time budget is hiding one of its most consequential variables.
VulcanBench’s July 26 report on Claude Opus 5 makes the mechanism concrete. The model’s lowest-effort setting was its best overall, solving 20 of 23 tasks. Its high-effort setting solved only 18. The extra reasoning was not useless in isolation: high effort returned the fewest wrong answers of any setting, just one against three at low effort. But it ran out of clock on tasks that the cheaper setting completed, and a timeout scores zero. Two of the three regressions were cutoffs on tasks low effort solves cleanly. Given unlimited time, high effort merely ties its cheapest setting while costing 3.1 times as much.
This has a direct consequence for anyone building a routing ladder, the common design where a cheap model attempt escalates to a more capable and more expensive model when it fails. The standard assumption is that the next rung is better and merely costs more. For a meaningful share of model and task combinations, that assumption is wrong. You pay the higher rung’s price and escalate into a timeout or a token cap rather than a correct answer.
Cost per successful task is becoming the metric that actually matters
Several independent groups have converged on the same alternative metric in recent months: cost per successful task, meaning total spend including all failed attempts divided by the number of tasks that passed an acceptance check. The convergence is the strongest signal that this approach is becoming standard.
VulcanBench reports dollars per solved task as a headline column. Long-Horizon-Terminal-Bench publishes per-task cost alongside accuracy, and its most instructive row is GPT-5.4 at roughly $26 per task with a much lower pass rate than Grok 4.5 at approximately $11. TestEvo-Bench runs agents under a cost cap, and Claude Code’s test-generation score falls from 71% to 44% at the tighter cap. The pattern across all three is the same: the model with the best raw accuracy score is rarely the model with the best cost-per-success ratio.
Vendors are already pricing products this way. HubSpot moved its Breeze Customer Agent in April to 50 cents per resolved conversation, down from $1 per handled conversation. Zendesk bills per automated resolution. Intercom’s Fin charges 99 cents per outcome and bills only on end-to-end resolution. The shift from paying for tokens to paying for outcomes is already happening at the product layer, and the benchmarking infrastructure is catching up.
What this means for teams deploying models now
There are four practical changes that follow from this analysis. First, every agent run should emit a failure reason as a required field, distinguishing budget exhaustion, verifier failure, and harness error rather than collapsing them into a single failure flag. Until you can separate a timeout from a wrong answer, your pass rate is measuring two different problems simultaneously and you cannot tell which one to address.
Second, compute cost per successful task at each effort level, not just per model. The ranking will not match the rate card, and the cheapest configuration may well win outright. Third, cap on tokens rather than wall-clock time unless latency is genuinely part of your service-level objective. A wall-clock cap scores your provider’s serving speed as though it were model quality.
Fourth, check the default effort setting on everything already deployed. Qwen 3.8-Max runs at its highest reasoning setting when the effort field is left unset, and its highest setting was its worst performer in independent testing. A team that never touches that parameter is running the configuration that costs the most per solved task.
The deeper point is that the AI industry is in a transition where the marketing layer, the benchmark layer, and the billing layer are measuring three different things and presenting them as though they were one. Teams that treat cost per successful task as a first-class metric, and that treat time and token budgets as explicit acceptance criteria rather than hidden footnotes, will make systematically better procurement decisions than those who rely on leaderboard rankings alone. The models are genuinely improving. The measurement frameworks are lagging behind, and that gap is where real money is being lost.
