Latency
Also called Time to first token, TTFT, Response time
How long an AI system takes to respond, often measured as time to the first token and then output speed in tokens per second.
What is latency?
Latency is the delay between sending a request to an AI model and getting the answer. For language models, it's usually described with two numbers:
- Time to first token (TTFT): how long until the first word appears.
- Output speed: how many tokens per second the model produces after that.
The total wait is roughly the first plus the length of the answer divided by the second.
How does it work?
Several things add to the delay: sending the request over the network, the model reading the whole input, waiting in a queue at busy times, and generating the answer one token at a time (inference). Longer prompts slow the first token; longer answers slow the finish. Reasoning models add thinking time before they reply, which can be seconds or minutes.
What is it commonly confused with?
- Throughput. Throughput is how many requests or tokens a system handles overall. A system can serve many users at once and still feel slow to each of them.
- Model quality. Faster isn't worse by default. Smaller models are often both quicker and good enough.
Why does it matter?
People notice delays quickly in chat, voice and autocomplete, so latency often matters as much as accuracy. Common ways to cut it: stream the answer so text appears as it's written, use a smaller model or route simple requests to one, ask for shorter answers, and reuse long, repeated prompt sections with prompt caching where the provider supports it.
