What is Latency?

The time a model takes to respond. Low latency matters for real-time chat and interactive apps.

Latency is how long you wait for a model's response. Two measurements matter in practice. Time to first token is the delay before any output appears, which drives how responsive a chat feels. Total generation time depends on output length and the model's tokens-per-second speed. Latency is affected by model size, whether it is a reasoning model, prompt length, server load, and where the model is hosted. Streaming helps perceived speed because text appears as it is generated. Providers often offer fast, small models (like Haiku or Flash tiers) alongside slower, smarter flagships. Techniques like prompt caching and quantization reduce latency, while long contexts and extended thinking increase it. For voice assistants and live chat, sub-second first-token latency is usually the target.

Example

A customer support chatbot that takes 8 seconds to start replying feels broken, so a team switches from a large reasoning model to a small fast model for routine questions and keeps the big model for escalations.

Why it matters

For anything interactive, latency shapes the user experience as much as answer quality. Always test response speed under realistic prompts before committing to a model or tool. Browse the AI tools directory or the model leaderboard to put it into practice.