Why enterprises must escape the token trap now

AI offers so many obvious benefits that nine out of ten organisations have rushed in. Many have already integrated AI-powered capabilities into existing services and workflows; others have created entirely new offerings and ventures. 

One reason behind this explosion is that frontier AI models make prototyping new ideas much faster, cheaper and easier than previous approaches, particularly for digital tools and software-based services. However, this creates a hidden trap that can quickly drive up organisations’ bills.

AI pilot projects are usually run in isolation, or in limited test deployments, where token costs are easy to manage. When projects move to full deployment across a workforce or customer base, token consumption rises with every new user and request, leaving companies with a hefty bill.

In addition, models run by external companies introduce a degree of dependency, where providers may retire existing models at any time, forcing teams to re-test and rework prompts and workflows. Relying on an external provider also means relying on its uptime; last month, for example, a power event in one of Google Cloud’s London zones caused a multi-service outage for customers across the UK, disrupting downstream AI services for multiple hours.

Data sovereignty is yet another consideration. Using a hosted model means sending prompts, and any business data they contain, to third-party infrastructure that may sit outside the company’s HD. For organisations in regulated sectors, or those handling personal data, this raises questions about where data is processed and who can access it.

The Inference Paradox: Falling prices, rising bills 

These are real concerns, but for most organisations the more immediate problem is cost.

Analyst firm Gartner calls the underlying dynamic the “Inference Paradox”. Token prices have been trending downwards in recent years, but more capable models, and AI agents that work through tasks in multiple steps, consume far more tokens than a simple chatbot exchange. And it’s happening at a rate that’s rising far faster than token prices are dropping. As a result, Gartner forecasts that the inference cost of each agentic workflow will rise more than 400% over the next two years. 

“Product leaders cannot rely on more efficient token economics to rationalise AI costs,” said Gartner’s Senior Director Analyst Will Sommer.

This can become an issue even with low per-token pricing. Take an internal assistant built on a frontier model where a typical request costs just under 2 cents. A pilot with 50 staff making 20 requests a day comes to around $18 a day. Roll it out to 5,000 staff and that becomes $1,800 a day, or roughly $650,000 a year.

This presents a certain irony: businesses want these AI tools to be widely adopted, but the more employees and customers come to depend on them, the bigger the bill.

Per-token pricing also leaves businesses exposed to price changes they have no control over. Any price increase is multiplied by the volume of tokens the business consumes, so the more embedded the tool becomes, the larger the impact. 

According to Gartner research on building trust in AI-native workplaces, only 21% of surveyed IT leaders believe AI pricing will be fair and predictable.

There is a precedent for this. Uber spent years subsidising fares with investor money to win customers, and once it turned towards profitability, prices rose sharply: average ride-hailing prices across Uber and Lyft climbed 92% between January 2018 and July 2021, according to data from Rakuten Intelligence. 

In a report on AI market consolidation published in September, Gartner said it expects AI to follow a similar path, with price subsidies through at least 2028 and providers only earning sustainable margins once the market consolidates. 

With this in mind, business cases built around today’s rates may not hold indefinitely.

Should businesses move AI in-house? 

To mitigate these risks, a growing number of organisations are turning to open-weight models that can run on their own servers. Unlike frontier models from companies such as OpenAI and Anthropic, which are only available running on the vendor’s cloud, open-weight models can be downloaded and run on a company’s own infrastructure.

These models, such as Meta’s Llama and Alibaba’s Qwen family, generally trail the leading frontier models on the most complex reasoning tasks, but the gap has narrowed. Smaller versions can even run on a single server.

Common uses include high-volume tasks that don’t need the most capable model available: summarising documents, transcription and translation, searching internal knowledge bases, and classifying or routing customer queries.

Spiky workloads – where demand can quickly scale up or down in sharp peaks – are still best suited to running in cloud environments which are configured to support rapid fluctuations in compute demand. However, stable AI workloads running at high volumes are well-placed to be run on-premise. 

“The choice of infrastructure model for AI inference must revolve around infrastructure utilisation,” said Max Smolaks, Senior Analyst at Uptime Institute. “Uptime Intelligence has found that if an organisation can reach consistent 25% GPU utilisation in a data centre that it owns, it could potentially beat major cloud providers on inference cost. It would take GPU utilisation of around 60% to beat the cost of inference-as-a-service.”

These figures, from Uptime’s March report “Where to deploy AI inference: a guide to the economics”, assume an organisation already has data centre capacity and in-house skills, and depreciates its hardware over three years.

In-house AI: Buy versus build

One of the biggest challenges with on-premise AI is the upfront cost of the infrastructure needed to support it. Even in existing data centres, dense GPU systems can require new power and cooling, although smaller models may run within existing facilities with fewer changes. 

One way around this issue is moving hardware costs from CapEx to OpEx via leasing. The business pays a fixed monthly or quarterly fee over an agreed term, typically between two and five years, through OEM finance arms, resellers or specialist lessors. Support and maintenance are often bundled in.

This makes costs predictable and avoids tying up cash in hardware while component prices remain high: analyst firm TrendForce expects server memory contract prices to keep rising every quarter until the second half of 2027. 

Once the system is running, costs are tied to the lease, power and staff time, so they no longer rise with every token processed. Many leases also include the option to refresh hardware at the end of the term, so organisations aren’t left running ageing GPUs as models become more demanding. 

There is a catch, however. Smolaks says that running open-weight models on-premises is “the cheapest path to enterprise AI, but this requires a level of expertise many organisations are currently lacking”.

Suppliers can help bridge this gap, working with organisations to build out a customised solution for their needs, including what hardware it’s based on, and what model it runs. Partners can help match compute, memory and storage allocations to the volume of requests an AI system has to handle, where its data must stay and the systems it needs to connect to, as well as the space and power available in existing facilities. This rightsizing helps organisations avoid paying for compute capacity that spends most of the time sitting idle.

Final decision

AI can be a powerful tool for delivering operational efficiency, but when used indiscriminately, it can wind up having the opposite effect. Businesses that have already rolled out AI systems may wish to audit their token spend to check whether they’re running stable, high-volume workloads that would cost less on their own hardware, and companies considering how to scale beyond pilot programmes should model what their potential long-term token bills could grow to.

As AI becomes more embedded in day-to-day operations, organisations that review now where each workload runs, and what it costs, will be best placed to keep spending under control.