LLM · STEP 2
What Are Embeddings?
An embedding is a vector, a list of numbers, that stands in for a token's meaning. After tokenization turns text into integer IDs (see what tokenization is), each ID is looked up in an embedding table and swapped for its vector, positioned in a high-dimensional space where tokens with similar meaning land close together and unrelated tokens land far apart.
From an arbitrary ID to a meaningful position
A token's integer ID is arbitrary, a fixed slot number with no relationship to meaning: token 318 isn't inherently "closer" to token 319 than to token 9000. The embedding table replaces that ID with a vector, often hundreds of numbers long, and that vector's actual values are learned during training so that tokens used in similar contexts end up with similar vectors.
This is what makes an embedding a meaning map rather than just a longer ID: "king" and "queen" land near each other, "Paris" and "France" land near each other in a different region, and a word like "banana" that shares no context with either sits far from both. Nobody hand-labels these positions; they fall out of the same training process (predicting masked or next tokens across huge amounts of text) that shapes the rest of the model.
Position gets added in too
A raw token embedding on its own carries no information about where in the sequence that token sits, but word order matters: "dog bites man" and "man bites dog" use the same tokens in a different order with a very different meaning. A positional signal, encoding each token's position in the sequence, is added to its token embedding, so the vector the model actually works with going forward carries both what the token is and where it sits.
Why this layer matters beyond LLMs
The same meaning-map idea, project something into a vector space where distance reflects similarity, is the mechanism behind semantic search, recommendation systems, and retrieval augmented generation (see what RAG is): a query gets embedded, and whatever's stored nearby in that space is judged relevant, using nothing but vector distance.
FAQ
- Are embeddings the same thing as a lookup table forever fixed after training?
- For a token's base embedding, yes, that table is learned once during training and stays fixed at inference time. But the vector a token actually carries by the time it reaches later layers keeps changing, reshaped at each layer by attention and the position signal, so "embedding" specifically refers to that first, static lookup step.
- How many numbers are in one embedding vector?
- It varies by model, commonly a few hundred to a few thousand dimensions. More dimensions can capture finer shades of meaning but cost more memory and compute, so the size is a deliberate design tradeoff, not a fixed standard.
Sources
Related
Last updated 2026-09-16