LLM · STEP 4
What Are Transformer Layers?
A transformer isn't one computation, it's the same block, attention followed by a small feed-forward network, repeated dozens of times in a stack, each layer feeding its output to the next. Stacking that one block is what turns a token's vector from a shallow signal about word shape into a rich representation carrying facts, relationships, and intent.
One block, run over and over
Each layer of a transformer does the same two things to every token's vector: attention (see what attention is), which lets the token pull in relevant information from the rest of the sequence, followed by a small feed-forward network that processes that token's vector on its own. The output feeds directly into an identical block above it, and a large model repeats this dozens of times, GPT-3-scale models run on the order of 90-plus layers.
Nothing about the block's structure changes from layer to layer. What changes is the vector passing through it: each pass refines it a little further, so the version of a token's vector at layer 50 carries meaningfully more context than the version at layer 1, even though both went through mechanically identical steps.
Different depths specialize in different kinds of information
Interpretability research probing what different layers actually represent finds a rough progression. Early layers pick up shallow, local patterns: word shape, basic grammar, which part of speech a token is. Middle layers pick up more concrete relational facts: who did what to whom, entities and their relationships. Later layers carry more abstract signal: tone, intent, the kind of higher-level structure needed to decide what a sentence is actually asking for.
This progression isn't hand-designed into the architecture; every layer runs the identical block. It emerges from training, because building useful late-layer representations for predicting the next token requires the model to have already resolved lower-level structure earlier in the stack, the same way understanding a sentence's tone requires first knowing what its words and grammar are.
FAQ
- Does adding more layers always make a model better?
- Not without limit, and not on its own. More layers add capacity and cost, but a model also needs enough training data and compute to actually make use of that extra depth. Beyond a point, other levers, data quality, training technique, architecture details, start mattering more than raw layer count.
- Why can't a model skip straight to the final layer's output?
- Each layer's output is only meaningful because it was built on the layer before it. A late layer's representation of tone or intent presumes the grammar and factual relationships earlier layers already resolved; there's no shortcut that reaches abstract understanding without passing through the more basic structure first.
Sources
Related
Last updated 2026-09-16