Trending Topics
Trending
Your AI ROI is built on a price you don’t control
This article is part of our Opinions section, where we invite industry professionals to share their views on the most pressing technology questions of our time.
OpenAI CFO Sarah Friar recently made a sensible argument about the way companies measure AI. Rather than obsessing over cost per token, businesses should look at “useful intelligence per dollar”. That means defining the task and its success criteria upfront, then measuring whether the AI completes it successfully and what each successful outcome costs.
I agree with that. Cost per successful outcome definitely tells you more than raw token count, though it still has a critical gap. The dollar side is not fixed. If I build an AI business case around today’s price per million tokens, I am making assumptions about somebody else’s infrastructure costs, utilisation, margins, and commercial strategy, and assuming those conditions will remain favourable for the useful life of whatever I am building.
These cost and pricing factors are where I think many AI ROI calculations become exposed to a future re-pricing event.
There is a real cost underneath every token
A token may feel like an abstract unit on an API bill, but producing it requires physical infrastructure: specialised processors, memory that moves model weights and context, electricity, cooling, networking, data centres, and the capital to acquire and operate them.
One detail stands out when examining cost. Processing a prompt and generating an answer place different demands on the hardware. During generation, performance is often constrained less by raw computing power than by the speed at which the system can move data through memory. Utilisation is also a factor. An expensive GPU costs money whether it is working flat out or sitting half idle, so identical hardware can produce very different economics depending on traffic, batching, and how effectively the provider keeps it busy.
I am not suggesting CFOs now need to become GPU engineers. What they do need to understand is that there is no single underlying “cost of a token”. The economics vary among providers long before a commercial team decides what price to set for an API.
The price tells you less than you think
One provider may use a more efficient model architecture. Another may own more of its infrastructure or design its own silicon. Others may be particularly good at managing their serving environment, while very large providers gain another advantage from keeping expensive infrastructure consistently utilised.
These are genuine cost differences, but customers do not see them directly. They see a price with commercial decisions layered over the top. A provider can pass an infrastructure advantage through to customers, keep more of it as margin, or deliberately price aggressively while it fights for market share. A competitor with a higher underlying cost can arrive at the same published price for completely different reasons.
There is also a difference between marginal cost and fully loaded cost. A provider may cover the immediate cost of serving another token while its published price tells you very little about whether it is recovering the much higher costs of model development, infrastructure, and a hardware fleet that continually needs replacing.
The scale of the infrastructure commitments sitting underneath these economics makes that distinction important. Nikkei Asia recently estimated that five major US technology companies had accumulated around $1.65 trillion in off-balance-sheet commitments, including leases and future hardware obligations associated with the infrastructure buildout. That is not proof that token prices are about to rise. Technology will become more efficient, and competition may continue pushing prices down.
The lesson is simply that today’s API price reflects technology economics and market strategy at a particular moment. It should not be treated as a permanent property of AI.
Your architecture determines how exposed you are
Once I started looking at the problem this way, I could then focus on what the buyer can control. You cannot control a model provider’s silicon strategy, hardware utilisation, or future pricing decisions, but you can influence how much model inference your own architecture requires to complete a task.
Much of an agent’s token consumption can come from finding the information needed to answer a question rather than producing the answer itself. Consider a fairly ordinary enterprise question: “Which of my renewal accounts currently have critical support issues?”
An agent without a reliable understanding of the underlying data may explore its way toward the answer. It retrieves information, reasons about what it found, searches again, discovers another relationship, and continues until it believes it has enough context. Every additional round consumes tokens and introduces another opportunity for the agent to take a wrong turn.
Yet much of that question is structured. Accounts, renewal dates, ticket severity, and their relationships already exist as data. A database can perform that join precisely on ordinary CPU infrastructure rather than asking a language model to rediscover the relationship through repeated inference. This can still be a challenge to an agent, though, as the data often exists in silos. Providing a map of the data dramatically improves the efficiency of the agent.
That does not remove the model. Natural language still needs to be interpreted, and enterprise questions frequently combine structured facts with unstructured information such as messages, support conversations, and documents. The opportunity is to stop making the large-language model (LLM) perform work that another part of the system can do more accurately and efficiently.
Predictability belongs in the ROI discussion
Reducing token consumption matters, but enterprise buyers also need to understand how costs behave as AI moves from pilot to production. An agent whose retrieval path changes substantially from question to question can create a variable cost profile, particularly if the amount of exploration increases as the underlying data grows. An application that looks economical during a small test may therefore behave very differently at enterprise scale.
This can be seen in enterprise AI benchmarking that examines how agent performance and costs change as the underlying data scales. In these tests, accuracy can remain broadly stable as the data grows by orders of magnitude, while the cost difference between an architecture that has to explore for an answer and one that already understands the data’s structure can widen significantly.
The point is not to crown one model as the winner. The more useful measure is whether a task succeeds and how much it costs in tokens as the surrounding data scales. This helps separate the factors an enterprise can influence from the provider economics it cannot. The model is only one part of the cost equation; the architecture used to get that model to the correct answer also determines how much intelligence needs to be bought to produce the outcome.
Architecture directly impacts switching costs
This leads to another outcome of putting the structured work in the right place. If the relationships, permissions, and deterministic retrieval live outside the foundation model, the model becomes a more swappable component of the architecture.
That matters in a market where model capabilities and economics are changing quickly. Moving between providers will still require evaluation and engineering work because models are not interchangeable, but there is an important difference between changing a model and having to re-engineer an application that has become deeply dependent on one provider’s interfaces and behaviour.
The same architectural decision therefore does two jobs. It reduces the amount of expensive, variable inference required to answer a question and limits how much of the application depends on one provider’s future pricing. The lever that controls your cost is also the lever that provides optionality.
That is the piece I would add to the “useful intelligence per dollar” conversation. Measuring outcomes per dollar is right, but the dollar in that ratio remains a moving target set by somebody else. The number you can govern much more directly is how much model inference you need to buy to produce the outcome in the first place.
So, when evaluating AI spend, do not ask only what it costs today. Ask what happens to the business case if that price changes, and how much of the cost your own architecture allows you to control.
The cheapest token is the one you never spend. And unlike a discount, no provider can quietly re-price it on you.
Related articles
About The Author
