How Does the Generation Loop Work?

Given "The capital of France is", one complete pass through the model produces "Paris". That word is appended to the context, making it "The capital of France is Paris", and the model runs again, in full, on that new, longer sequence to produce the next token, a comma. Append again, run again, and the next token might be "a". Nothing is cached in the sense of skipping computation for old tokens' final answer; every token in the growing context gets reprocessed by attention at every new step (though key/value caching, an engineering optimization, avoids redoing every layer's work from scratch for tokens already seen).

This is why an LLM response streams out piece by piece rather than appearing all at once: each visible token really is the result of a separate full pass through the model, not a display effect layered over an answer that was already fully computed.

Generating a hundred-token answer means running the model roughly a hundred times, each pass a bit more expensive than the last as the context grows (see what attention is for why cost scales with sequence length). That's the direct reason response length is one of the biggest cost and latency levers in a production system, and why techniques that generate several tokens per pass (speculative decoding, among others) are an active area of engineering work specifically to cut down how many full loop iterations a response needs.

It also means the model commits to each token before it has any idea what it will write next. There's no lookahead or revision built into the base loop: once a token is chosen and appended, it's part of the fixed context every later step conditions on. Whatever looks like planning or backtracking in a model's output is either baked into patterns learned during training or added by an outer system (an agent harness re-prompting the model, or explicit reasoning steps), not a capability of this loop itself.