Inference Cost
Also called Token pricing, Cost per token, AI API pricing
What it costs to run an AI model on each request, usually priced per million input and output tokens through an API.
What is inference cost?
Inference cost is the price of using a trained AI model, as opposed to the cost of training it. Each time a model reads a prompt and writes an answer (inference), it uses computing power, and API providers pass that on, usually as a price per million tokens.
How does it work?
A typical bill has a few parts:
- Input tokens: everything you send, including the system prompt, conversation history, documents and tool results.
- Output tokens: everything the model writes, usually priced several times higher than input.
- Reasoning tokens: reasoning models often bill their hidden thinking as output, even though you don't see it.
- Discounts: many providers charge less for repeated prompt sections (prompt caching) and for batch jobs that can wait.
A simple estimate is: (input tokens × input price) + (output tokens × output price), multiplied by your number of requests.
What is it commonly confused with?
- A subscription price. A monthly chat plan hides token costs behind usage limits. Inference cost is what you pay when building on the API.
- Price per word. Prices are per token, and the number of tokens depends on the model's tokenizer.
- Free local models. Running an open-weight model yourself has no per-token fee, but you pay for the hardware and electricity instead.
Why does it matter?
Costs grow with every user and every message, and long conversations get more expensive as history builds up in the context window. The biggest savings usually come from using a cheaper model where it's good enough (model routing), trimming what you send, caching repeated content and limiting answer length.
