Trending Topics
Trending
Time to stop treating every new AI model as an upgrade
This article is part of our Opinions section, where we invite industry professionals to share their views on the most pressing technology questions of our time.
The other morning I joined a call a minute late because my laptop was tied up running a benchmark. I had kicked off a large run before going to bed, and it was still analysing the results.
That has become fairly normal. A new model appears and, almost immediately, somebody wants to know whether we have benchmarked it yet. Is it better than the model released last month? What does it cost? Should we be using it?
I understand the instinct. Model capabilities are improving quickly enough that ignoring new releases would be foolish. But I am increasingly uncomfortable with the assumption buried in the question: if a new model takes the lead, should an enterprise try to move to it?
We have created a market in which there is always another frontier to chase. The challenge for an organisation is working out when moving that frontier actually matters to the work it needs done.
The leaderboard is already becoming less useful
One way the AI industry is trying to make sense of this is through the Pareto frontier, an economics concept that is increasingly appearing in model comparisons.
Rather than simply ranking models according to intelligence, a Pareto frontier considers capability alongside cost. A model sits on the frontier when there is no other option that is both cheaper and at least as capable. That allows several models to be sensible choices because buyers are optimising for different things.
Artificial Analysis recently used this approach when assessing Grok 4.6, placing it on its intelligence-versus-cost-per-task Pareto frontier. Its point was not simply that Grok had become one of the most capable models. The interesting part was the capability available for the price.
I like this way of thinking because it moves us away from the idea that there can be one “best” model and toward acknowledging that a model makes sense for a particular workload at a particular cost.
What I benchmark is bigger than a model
The public conversation often treats a model as though it were something you can swap in and out of production without disturbing anything around it. That is not my experience.
Before I can even benchmark a newly released model, I need to know how I am going to run it. Is the model available through the inference provider I am using? Does it work properly with the harness? Can it call the tools the task requires? What happens when it encounters the permissions, context, and workflow we have given it?
A production AI system has a lot more around the model than the model name on the API call.
DeepSeek recently made this unusually explicit when it released its DeepSeek Harness developer preview. Its description is wonderfully simple: “Agent = Model + Harness.” The harness deals with the environment around the model, including its tools, sessions, storage, and the loop that keeps the agent working.
In practice, that means you are never really testing a model in isolation. You are testing how that model behaves inside a particular operating environment.
I have seen this in our own testing on Enterprise-Bench. One recent model we tested was deliberately designed to be highly independent on long-running tasks. That is an interesting capability. Put it into a tightly controlled enterprise workflow, however, and that independence can become a problem. We saw it question instructions and try to decide whether to take another path when the task required it to follow an established process.
The model was doing what it had been designed to do. Our workload wanted different behaviour. A benchmark should narrow the field, not make the deployment decision. A strong result tells me a model deserves a closer look. It cannot tell me that the model, the provider, and the operating environment will improve the work my organisation needs done.
An easy API change can still be a difficult enterprise change
There is another reason I would be wary of treating every new frontier model as an upgrade opportunity. Enterprise systems have accumulated processes around change for good reasons.
If somebody releases a new version of an important application, we do not normally deploy it across the entire organisation the next morning. We test it against our environment. We give it to a smaller group. We look for failures. Security and governance teams need to understand what has changed. Only then does it move further into production.
We already know how to manage this kind of change. AI should not get a free pass simply because the release cycle is faster. Some of the technical barriers to switching models are getting lower. DeepSeek, for example, documents an Anthropic-compatible endpoint for using its models with Claude Code. That can make experimentation much easier.
It doesn’t tell me whether the new combination will behave the same way under production traffic, respect the workflow the same way, use tools as reliably, consume the same amount of context, or fail in the same places. Those differences may be manageable in a test environment. In production, somebody has to own them.
Changing the model can ripple through far more of the system than the API call suggests. You may have to adjust prompts and tool behaviour, rerun the evaluations that gave you confidence in the first place, and put a new provider through security or procurement. By the time the model reaches production, several teams may have had to do real work simply to prove that the apparent improvement survives contact with your environment.
Then another model comes out three weeks later.
Better is not always worth changing for
This is where I think enterprise leaders need to push back on some of the urgency coming from the AI market. The threshold for changing a production system cannot simply be that the latest model scored higher – it must be better enough.
Better enough on the workloads you care about to justify the testing, operational work, and disruption involved in moving to it.
That threshold will be different for every enterprise and every use case. A substantial improvement on a difficult task that is currently limiting what you can automate may justify a quick change. A few additional benchmark points probably will not.
The frontier is going to keep moving. That does not mean the enterprise has to move with every change. A benchmark can tell you when a new model is worth testing. What happens after that depends on whether the improvement is significant enough to justify changing what is already running.
More from our Opinions section
About The Author
