What is the difference between a prompt and a system prompt?

The prompt is everything the model is given to continue. The system prompt is one part of it: instructions from the person who built the app, such as tone or rules, placed ahead of your message. Providers expose it differently. Anthropic uses a separate system field, and OpenAI uses a developer role that ranks above user messages.

Why does the way I ask change the answer?

The model continues whatever text it is given, so different wording is a different starting text with different likely continuations. More specific wording narrows what fits. Models can also react to small formatting changes, and the same prompt can give different answers on different runs.

What should a good prompt contain?

Always a clear task: an action verb saying what you want done. Context and examples make the answer better by narrowing it, and a persona, a format and a tone tune how it reads. The task is the part you cannot skip, and the rest are optional layers to add when the first answer misses. This ranking is a rule of thumb, not a measured result.

Is prompt engineering the same as fine-tuning?

No. Prompt engineering edits the input and leaves the model's weights untouched. Fine-tuning trains the model further on new examples and changes the weights. Few-shot examples in a prompt change nothing about the model itself.

Does "think step by step" always work?

No. Kojima et al. measured large gains for InstructGPT (text-davinci-002) on arithmetic benchmarks in 2022, but that is a result for specific models and tasks. Whether a given phrase helps a given model on a given task has to be tested.

Does the model remember my earlier messages?

Not by itself. In Anthropic's API, no conversation state is kept between requests, so the application includes the earlier turns in each new prompt. Whatever is not in the prompt, the model cannot see.

LLM · INPUT

What Is a Prompt? How AI Reads Your Instructions

A prompt is the text a language model is given and asked to continue. Everything the app sends, your message, the earlier turns, the builder's instructions and any tool results, is assembled into one sequence of tokens, and the model predicts the next token again and again. Changing the wording changes the sequence, so it changes the answer.

One-pager titled "What is a prompt?" with the subtitle "The text you send to the model, rebuilt every turn". A row of four boxes joined by plus signs, bracketed and labelled "The prompt: everything sent in": System ("main rules"), User ("your message", amber), Assistant ("past answer") and User ("new message", amber). An arrow leads to a Model box and on to a dashed Assistant box ("new answer"). A return line from the new answer back to the past answer is labelled "saved, then sent back next turn". An arrow from the first User box leads to a colour-coded funnel, widest first, with a downward arrow beside it linking each layer to its rank: Task (mandatory), Context and Examples (important), Persona, Format and Tone (nice to have). At the bottom right a box labelled "Example prompt" shows one realistic prompt in six colours that match the funnel: "You are a support lead." (persona), "Can you summarize this customer email:" (task), "[content of the email]" (context), "I want three bullet points" (format), "in a neutral and short tone." (tone) and "This is an example:" followed by three bullets, "Late order", "Wants a refund" and "Needs a reply today" (examples). — One-pager: the prompt is everything sent in (system, your message, the assistant's past answer and your new message), the model writes a new answer that is saved and sent back next turn, and a user message is written in layers: task, context and examples, persona, format and tone.

A prompt is the text the model continues

A prompt is the input a language model receives. It is text, and the model's job is to continue it. Before the model sees anything, the text is cut into tokens, and the model then predicts the next token, one at a time.

That holds for a chat assistant too. Hugging Face's documentation on chat templates states the central point: all causal language models, chat-trained or not, continue a sequence of tokens, and a chat is still a sequence of tokens.

So a prompt is less a command than the opening of a document. The reply is a likely continuation of it. Training on conversations is what makes that continuation look like an answer. The InstructGPT paper (OpenAI, 2022) found that making a model bigger does not by itself make it better at following what a user wants, which is why such models get extra training on human feedback.

Roles: how a conversation becomes one prompt

Chat apps and APIs describe a prompt as a list of messages, each with a role. The user role is what you type. The assistant role is what the model wrote earlier. A third role carries instructions from the person who built the app, such as the tone to use or the rules to follow.

The vendors differ on the details. Anthropic's Messages API takes those instructions in a separate top-level system field and has no system role inside the message list. OpenAI's current docs use a developer role and say developer messages take priority over user messages. Open-weight models ship their own chat templates: Hugging Face shows Mistral-7B-Instruct wrapping user turns in [INST] markers where Zephyr-7B uses <|user|> and <|assistant|>.

Behind any of these shapes, a final step flattens the list into one sequence of tokens with role markers inside it. The model does not receive separate inputs. The earlier turns are in that sequence too: Anthropic's API keeps no conversation state between requests, so the application sends the previous turns again each time. Tool results are added the same way, as a tool_result block in Anthropic's API and a function_call_output item in OpenAI's, and tool descriptions count as input tokens. All of it has to fit in the context window.

Why wording changes the answer

The next token depends on every token before it. Change a word in the prompt and you change the context that the probabilities are computed from. Adding detail narrows the range of continuations that fit. "Write something about dogs" is compatible with a poem, a fact sheet or a story. "Two plain sentences on adopting a rescue dog" rules most of those out. Both prompts are examples, and the figure is a schematic of the idea, not a measurement.

Models can also react to changes that look trivial. Sclar and colleagues found that formatting differences in few-shot prompts, such as separators and spacing, moved accuracy by up to 76 points for LLaMA-2-13B in the settings they tested, and that a format that worked well for one model correlated only weakly with how it did on another.

The same prompt can also give different answers on different runs, because the next token is sampled rather than always taken from the top of the list. OpenAI's prompting guide describes model output as non-deterministic. How that sampling works is covered on the next-token prediction page.

Building a prompt: always a task, then layers that improve it

A practical way to write a prompt is in layers, from the part you cannot skip to the parts that polish. Every prompt needs a task: an action verb such as write, rephrase, summarize or compare that says what you want done.

Context and examples make the answer better. Context is the data the model works from, plus just enough background to narrow the possibilities, as in the dogs example above. An example shows the output you want, which is the few-shot idea covered in the next section. Both are optional, and both usually make the result closer to what you meant.

Persona, format and tone tune the answer even more. A persona asks the model to take a perspective, such as a teacher or a data analyst. Format fixes the shape of the answer, such as a table or bullet points. Tone sets the register, such as formal or casual. They change how the answer reads more than what it says.

The model has no common sense about your situation, so a constraint you leave out is not assumed. "Rephrase this text" leaves open what to keep. "Rephrase this text without changing any technical term, in shorter sentences" rules out most of the rewrites you would reject. A complete prompt can use every layer: an expert editor as the persona, a request to write an introduction as the task, the pasted text as the data, a sample of the output you want, a plain scientific tone, and a format such as keywords with definitions followed by eight lines.

The ranking is a rule of thumb, not a measured result. Which layers matter most depends on the model and the task, so a template is a starting point to adapt and test on your own case. Both Anthropic's and OpenAI's prompting guides describe prompts as built from the same kinds of parts: instructions, examples, extra context and a role.

Few-shot examples and step-by-step prompts

Putting worked examples in the prompt is called few-shot prompting. The GPT-3 paper (Brown et al., 2020) showed a model picking up tasks from demonstrations given purely as text, with no gradient updates or fine-tuning. Nothing about the model changes. Only its input does. Anthropic's guide calls examples one of the most reliable ways to steer output format, tone and structure.

Chain-of-thought prompting applies the same move to reasoning. Wei et al. (2022) put examples with intermediate steps in the prompt, and large models did better on arithmetic, commonsense and symbolic reasoning tasks. Kojima et al. (2022) then found that adding "Let's think step by step" with no examples raised InstructGPT (text-davinci-002) accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7%.

Those are results for specific models and benchmarks from 2022. They are not a rule that the phrase helps every model on every task.

What prompt engineering is, and where it stops

Prompt engineering is the practice of editing the context so the likely continuation is the one you want: clearer instructions, examples, the right documents, a stated output format. OpenAI's guide defines it as writing effective instructions for a model and calls it a mix of art and science. It changes the input. It does not change what the model learned, which is what fine-tuning does.

It has limits. Output varies between runs, and results can shift when a provider updates a model. That is why OpenAI recommends pinning production applications to specific model snapshots and building evaluations that measure prompt behavior. No wording is guaranteed to work.

A system prompt is also not a security boundary. It is more text in the same sequence, so a later message, or a web page the model reads, can contain instructions that compete with it. OWASP's guidance says a system prompt should not be considered a secret, nor used as a security control. That attack is covered on the prompt injection page.

FAQ

What is the difference between a prompt and a system prompt?: The prompt is everything the model is given to continue. The system prompt is one part of it: instructions from the person who built the app, such as tone or rules, placed ahead of your message. Providers expose it differently. Anthropic uses a separate system field, and OpenAI uses a developer role that ranks above user messages.
Why does the way I ask change the answer?: The model continues whatever text it is given, so different wording is a different starting text with different likely continuations. More specific wording narrows what fits. Models can also react to small formatting changes, and the same prompt can give different answers on different runs.
What should a good prompt contain?: Always a clear task: an action verb saying what you want done. Context and examples make the answer better by narrowing it, and a persona, a format and a tone tune how it reads. The task is the part you cannot skip, and the rest are optional layers to add when the first answer misses. This ranking is a rule of thumb, not a measured result.
Is prompt engineering the same as fine-tuning?: No. Prompt engineering edits the input and leaves the model's weights untouched. Fine-tuning trains the model further on new examples and changes the weights. Few-shot examples in a prompt change nothing about the model itself.
Does "think step by step" always work?: No. Kojima et al. measured large gains for InstructGPT (text-davinci-002) on arithmetic benchmarks in 2022, but that is a result for specific models and tasks. Whether a given phrase helps a given model on a given task has to be tested.
Does the model remember my earlier messages?: Not by itself. In Anthropic's API, no conversation state is kept between requests, so the application includes the earlier turns in each new prompt. Whatever is not in the prompt, the model cannot see.

Sources

Last updated 2026-10-11