Test-Time Compute
Also called Inference-Time Compute, Inference-Time Scaling
Extra computation a model spends while answering, such as thinking through a problem or trying several answers, to improve the result without retraining.
What is test-time compute?
Test-time compute is the amount of computation a model uses while it is answering a question, rather than while it is being trained. Spending more of it, for example by reasoning step by step or by generating several candidate answers and picking the best, can improve accuracy on hard problems.
How does it work?
Instead of answering immediately, the model is given room to think. Common approaches include longer chain-of-thought reasoning, sampling many answers and choosing the best, and having the model check and revise its own work. Reasoning models are built around this idea.
What is a simple example?
On a hard maths question, a model that writes out its steps and checks them before answering is more likely to get it right than one that answers in a single pass.
Why does it matter?
A 2024 study found that, on moderately hard problems, a smaller model given extra test-time compute could beat a model 14 times larger under matched compute. The trade-off is that each answer costs more and takes longer, which raises inference cost and latency.