How Does Next-Token Prediction Work?

The final layer's output vector for the last position in the sequence is multiplied against the model's full vocabulary matrix, producing one raw score per token in the vocabulary. Those scores are converted into a proper probability distribution, values between 0 and 1 that all sum to 1, via a softmax function. For the prompt "The capital of France is", a well-trained model puts the overwhelming majority of that probability mass on "Paris", with a long tail of small probabilities spread across everything else.

Always picking the single highest-probability token (greedy decoding) makes output deterministic but often flat and repetitive. Instead, generation typically samples from the distribution, weighted by those probabilities, so the top candidate is most likely to be chosen but not guaranteed every time. Two controls shape how that sampling behaves: temperature reshapes the distribution before sampling, low temperature sharpens it toward the top candidates (more predictable, more repetitive at the extreme), high temperature flattens it (more varied, more prone to picking a weak candidate). Top-p (nucleus) sampling instead restricts sampling to the smallest set of top candidates whose probabilities sum past a threshold, cutting off the long tail of implausible options regardless of how the distribution is shaped.

The model has no separate mechanism for checking truth; it only ever asks "which token is statistically likely to come next, given everything the training data showed for similar contexts." When the true answer was well-represented in training, that statistical likelihood and factual correctness line up closely. When it wasn't, the model still produces a probability distribution and still samples fluently from it, which is exactly why hallucinated text so often reads as confident and grammatically clean: fluency and correctness are computed by two different things, and only one of them is what this step actually optimizes.