LLM · STEP 3
What Is Attention in a Transformer?
Attention is the core mechanism inside a transformer that lets every token look at every other token in the input and decide how much each one should matter to it. It's how a model figures out what a pronoun refers to, or which earlier word gives a later one its specific meaning, regardless of how far apart they sit in the text.
Every token asks: what here matters to me?
Take the sentence "The animal didn't cross the street because it was tired." Nothing in the word "it" says what it refers to; that has to come from context. Attention computes, for the token "it", a weight against every earlier token in the sentence, how relevant is "animal", how relevant is "street", how relevant is "tired", and uses those weights to blend information from the tokens that matter most into "it"'s own updated representation.
Mechanically, each token produces three vectors from its embedding: a query (what am I looking for), a key (what do I offer), and a value (what information do I actually carry). A token's query is compared against every other token's key to produce a relevance score, those scores are normalized into weights that sum to one, and the token's new representation becomes a weighted sum of every other token's value, weighted by those scores.
Why this beats reading strictly left to right
Older architectures processed text sequentially, one token affecting the next only through a compressed running summary, which made it hard to carry a detail from far back in a long input all the way to where it was needed. Attention connects any two tokens directly, in one step, regardless of distance, which is why a transformer can tie a pronoun to a noun several sentences back as easily as to the word right before it.
Multiple attention heads run in parallel, each learning to track a different kind of relationship (one head might specialize in subject-verb agreement, another in coreference like the "it" example), and their outputs are combined, giving the model several independent views of how the tokens relate to each other at once.
The cost this mechanism carries
Computing a relevance score between every pair of tokens means the amount of computation grows with the square of the input length: double the input, and attention costs roughly four times as much. That quadratic cost is the direct reason long-context models need real architectural work, not just a bigger number in a config file, to stay usable at tens or hundreds of thousands of tokens of input.
FAQ
- Does attention only look backward at earlier tokens?
- During generation, yes: a model predicting the next token can only attend to tokens that came before it (causal, or masked, attention), since the tokens after it don't exist yet. During training on existing text, or in some non-generative uses of transformers, attention can look both directions across the full input.
- Is attention the same thing as the transformer architecture?
- No. Attention is one mechanism inside a transformer block; a full block also includes a small feed-forward network applied to each token afterward, and transformer layers (see what transformer layers are) stack many of these blocks to build up meaning progressively.
Sources
Related
Last updated 2026-09-16