6 hours ago
8
AI and ML
Cheap can be expensive
When it comes to AI services, you don't necessarily get what you pay for.
It turns out that AI models with expensive tokens may cost less than models with cheap tokens for particular tasks. And the tooling attached to those models can have a significant effect on cost and output quality.
Databricks, which sells data analytics software and services, recently devised an internal coding benchmark to assess the tradeoff between price and performance using various AI models.
Matei Zaharia, CTO of Databricks and associate professor of computer science at UC Berkeley, said the company undertook the evaluation because models are often tuned to existing benchmark tests like SWE-Bench – which is "broken," according to OpenAI.
Databricks devised its benchmark using real engineering tasks performed by its staff to assess how AI agents perform. Zaharia said while the results reflect the company's internal codebase, other companies should be able to conduct similar evaluations using their own code.
One of the things Databricks found was that open weight models like Z.ai's GLM 5.2 are competitive with frontier models, like Anthropic's Opus 4.8.
"It landed in the top capability tier, statistically tied with Opus 4.8 on quality, but costing $1.28/task against Opus’s $1.94," the company said in its report.
But the price-per-token doesn't tell the whole story. Databricks contends that price-per-task needs to be considered.
"Cheaper per-token does not imply cheaper per-task," said Zaharia in a social media post. "For example, Sonnet 5 costs less per token than Opus 4.8 but used more tokens, resulting in higher cost and lower quality."
So while Anthropic's Sonnet 5 was around 1.7x cheaper than Opus 4.8 on a per-token basis, it was more costly on a per-task basis – $2.09 for Sonnet 5 compared to $1.94 for Opus 4.8. That's because it completed tasks less often (81 percent compared to 87 percent), and consumed more tokens to achieve the desired result.
Academics already reached this conclusion, noting back in March that in about a third of the model comparisons they conducted, the model with the lower listed price ended up costing more. "For example, Gemini 3 Flash's listed price is 80 percent cheaper than GPT-5.4's, yet its actual cost across all tasks is 38 percent higher," they observed.
The other thing that had a significant impact on test results was the harness – software like Claude Code, OpenAI Codex, and the Pi coding agent – which passes user input to the model, invokes various tools, and returns results.
"Harnesses make a huge difference in cost-performance," said Zaharia. "The very simple Pi harness got the same success rate as harnesses from the LLM vendors with Opus and GPT 5.5, but at 2x less cost!"
Zaharia attributed the difference to the size of the input – the context – passed to the model with every turn. When Claude Code served as the harness for Opus 4.8, Databricks measured a context of 742,000 tokens per task, compared to 236,999 for Pi. That's about 3.2x fewer tokens overall.
With Codex, the total context per task was 1,235,000 tokens, compared to 665,000 tokens for Pi, which is known for its minimal system prompt.
Zaharia said the results explain why Databricks built a tool called Omnigent to harness the harnesses – it's a wrapper for combining and swapping multiple coding agents. It's the front-end equivalent of the kind of back-end model swapping that OpenRouter enables. ®
English (US) ·