What Is an LLM (Large Language Model)?

Text is broken into tokens, roughly word-pieces rather than whole words. The model reads a sequence of tokens and outputs a probability distribution over the next one. Generation samples from that distribution, one token at a time, feeding each new token back in to predict the one after it.

Training hides the next token in a huge text corpus and makes the model guess it, nudging its weights, billions of numbers, every time it guesses wrong. Nothing labeled 'understanding' gets programmed in. The ability to answer questions or follow instructions falls out of that guessing game once the scale and the training data are large enough.

Nearly every current LLM is a transformer, and its core mechanism is self-attention. For each token, the model works out how much every other token in the input should shape its representation. That is how it ties a pronoun to a noun five sentences back, or a question to context anywhere in the input, no matter the distance.

Attention is also why context length is expensive. Naive self-attention cost grows with the square of the input length. Long-context models take real architectural and engineering work to get around that. You cannot buy the capability by raising a number in a config file.

A base model trained only on next-token prediction over internet-scale text is good at continuing text. It is not good at following instructions or turning down a harmful request. Instruction tuning and reinforcement learning from human feedback (RLHF) are later training stages that turn that base model into something that acts like an assistant. Pretraining supplies the raw capability. The later stages supply the behavior.