What is Token?

The unit of text an AI model processes — roughly 0.75 words in English. API pricing is charged per million tokens of input and output.

A token is the chunk of text a language model actually reads and writes. Before processing, your input is split by a tokenizer into these pieces, which can be whole words, parts of words, punctuation, or spaces. In English, one token averages around four characters, or roughly three quarters of a word, though code, rare words, and non-English languages often use more tokens per word. Tokens matter in two practical ways. First, pricing: API providers charge per token, usually quoted per million, with separate rates for input and output. Second, limits: a model's context window is measured in tokens, so a long document may not fit. Output speed is also often reported in tokens per second. Most providers offer a tokenizer tool so you can count tokens before sending a request.

Example

The sentence "AI tools are getting cheaper" is about six tokens. A 10-page report might be 5,000 tokens, which sets what you pay to have a model read it via an API.

Why it matters

Token counts drive both your API bill and whether your documents fit in a model's context window, so comparing per-token prices is the standard way to compare model costs. Browse the AI tools directory or the model leaderboard to put it into practice.