Trending Articles
Trending Topics
Trending
A new piece of research from Microsoft has concluded that throwing more computing resources at AI language models doesn’t necessarily generate better results.
The Microsoft researchers examined the performance of several prominent LLMs, such as GPT-4o, Claude 3.5 Sonnet and Gemini 2.0 Pro. It also included so-called reasoning models such as OpenAI’s o1 and DeepSeek R1.
They tested models using three different techniques.
First, standard chain-of-thought, where the AI works through answers step by step. The second used parallel scaling, where the AI generates multiple answers to the same query and aggregates those answers to arrive at a final solution.
Third came sequential scaling, where the model generates an answer and then uses feedback from a “critic” to refine the answer. In this research, the team used the same LLM to “critique” its own answer.
The researchers found huge variations between the amount of tokens used by different models to achieve roughly equivalent levels of accuracy. For instance, DeepSeek R1 consumed ten times as many tokens as Claude 3.7 Sonnet, but its accuracy is “slightly lower”, according to the researchers.
Tokens – how much text the model processes and generates – can be roughly equated to compute. The more tokens a model burns through the more computing resources are generally required.
While the team found that all models improve their accuracy with increased inference-time compute, those benefits are task-dependent and the improvements start to diminish with highly complex problems.
“More difficult problems requires more tokens,” the report states. “However, more tokens do not necessarily mean higher accuracy even for test-time scaling models, probably due to increase in difficulty levels.”
Even repeating the same queries on the same model can incur wildly different costs. “Repeated queries to the same model can yield highly variable token usage, introducing cost nondeterminism for developers and users – even when the model consistently provides correct answers,” the research paper states.
That presents obvious problems for companies trying to keep a lid on how much they’re spending on AI compute, with little predictability about the cost of running the same type of queries.
However, the Microsoft team expects that accuracy will continue to improve. “Encouragingly, all models demonstrate significant gains when inference is further scaled with perfect verifiers or strong feedback, suggesting ample potential for future improvements.”
