What is Transformer?

The neural network architecture behind modern LLMs, using an 'attention' mechanism to weigh the importance of different parts of the input.

The transformer is the neural network architecture behind nearly all modern language models, introduced in the 2017 Google paper "Attention Is All You Need". Its core idea is self-attention: for every token in the input, the model weighs how relevant every other token is, which lets it track relationships across long stretches of text (like connecting a pronoun to a name several paragraphs back). Transformers process tokens in parallel rather than one at a time, which made training on huge datasets practical and unlocked today's scale. The same architecture, with variations, powers text models (the "T" in GPT stands for transformer), vision models, and speech models. Practical model properties you deal with daily, like context window limits and per-token pricing, trace back to how transformers process sequences of tokens.

Example

When you ask a model "Sara handed the report to her manager because she was leaving early", attention layers help it work out which person "she" most likely refers to by weighing the surrounding words.

Why it matters

You do not need to understand transformers deeply to use AI, but knowing the basics explains why models have token limits, why long inputs cost more, and what vendors mean when they describe architectural improvements. Browse the AI tools directory or the model leaderboard to put it into practice.