What is Inference?

The process of running a trained model to generate output. Inference cost and speed (latency) are key practical factors when choosing a model.

Inference is what happens when a trained model is actually used: it takes your input and computes an output, one token at a time for language models. Training happens once (and costs the provider enormous sums); inference happens every single time anyone sends a request, so it is where the ongoing cost and speed of AI live. Inference performance is measured in a few ways: time to first token (how quickly the response starts), tokens per second (how fast it streams), and cost per million tokens. Providers reduce inference cost with techniques like quantization, batching, mixture-of-experts routing, and prompt caching. For open-weight models, you can run inference on your own hardware, trading setup effort for control and privacy. When people compare models on price and latency, they are comparing inference.

Example

A chatbot handling 10,000 customer conversations a day pays inference costs on every message. Switching from a large model to a smaller one for simple questions can cut that bill sharply.

Why it matters

Inference cost and speed determine what a tool costs to run at scale and how responsive it feels, often more than raw model capability does. Browse the AI tools directory or the model leaderboard to put it into practice.