What does RLHF stand for?

Reinforcement learning from human feedback. People compare a model's answers, a reward model learns those preferences, and the language model is tuned toward answers the reward model scores higher.

Why does ChatGPT agree with everything I say?

Not every time, but the pattern has a name: sycophancy. A 2023 study of five assistants whose training used human feedback, including GPT-4 and Claude 2, found it in all of them, and found that answers matching a user's views were more likely to be preferred in human preference data. How any product behaves today depends on its vendor's training, which that study does not describe.

Does RLHF make a model truthful?

Not by itself. The reward predicts what raters prefer. The InstructGPT paper reports improved truthfulness in its evaluations, and its authors also say the models still make up facts.

What is the difference between RLHF and fine-tuning?

Fine-tuning is the general term for continuing training on new data. Instruction tuning is fine-tuning on example answers to copy. RLHF is another kind, which learns from comparisons between answers instead of from answers to copy.

What is AI alignment, and is RLHF the same thing?

Alignment means getting a model's behavior to match what its users and builders intend. RLHF is one technique for that, not the whole field. The InstructGPT paper describes models that miss what users intend as not aligned with their users, and uses RLHF to move them closer.

LLM · TRAINING

What Is RLHF? How Chatbots Learn What People Prefer

RLHF, short for reinforcement learning from human feedback, is a training stage where people compare pairs of a model's answers and the model is tuned toward the ones they preferred. It shapes how a chatbot sounds and what it will decline. It learns what raters prefer, which is not always what is true.

One-pager titled "What is RLHF?" with the subtitle "People pick the better answer. The model is tuned toward it." Three steps left to right. Step 1, People pick: for the prompt "Why is the sky blue?", Answer A "Air scatters blue light." is marked Preferred and highlighted in amber, and Answer B "It is blue. Next question." is marked Passed over; the caption reads many prompts, many raters. Step 2, Reward model learns: a score bar for each answer, a long amber bar for A and a short steel bar for B; the caption reads then scores new answers. Step 3, Model is tuned: answers like A are marked more likely and answers like B less likely, and a dashed arrow from Before to After is labeled stays close. A REMEMBER band reads: preference, not truth; raters' taste sticks; leash limits the drift. The prompt and answers are an invented example. — One-pager: two answers to one prompt, the one a person prefers, and the model tuned toward it.

Where RLHF sits in training

A chatbot is built in stages. Pretraining teaches a model to continue text. Instruction tuning then shows it example questions with good answers. Both are covered on the LLM page, so this page starts after them.

RLHF is the stage that comes next, and it works differently. Nobody hands the model a correct answer to copy. People only say which of two answers they prefer, and the model is tuned toward the preferred one.

The idea is older than chatbots. In 2017, researchers at OpenAI and DeepMind taught game-playing and simulated-robot agents from people comparing short video clips, with feedback on less than 1% of the agent's interactions. Applied to language models, it became the method behind InstructGPT, which OpenAI described in 2022.

Collecting preferences and training a reward model

It starts with a prompt. The model writes two or more answers. A person, called a rater or labeler, picks the answer they prefer or ranks them. For InstructGPT, labelers ranked between 4 and 9 answers per prompt, and each ranking was broken into pairs. OpenAI hired about 40 contractors for the labeling work.

No team can rate every answer a model might write. So a second model learns to stand in for the raters. This reward model takes a prompt and an answer and returns one number, a score. It is trained so the answer a person preferred scores higher than the one they passed over. Once trained, it can score answers no person ever saw.

The prompt and answers in the figure are an invented example that shows the shape of the data. Real comparisons come from many prompts and many raters.

Tuning the model against the score, on a leash

Now the language model itself is trained. It writes an answer, the reward model scores it, and the model's weights are nudged so answers like the high-scoring ones become more likely. This repeats across many prompts. The score plays the role of the reward in reinforcement learning, which is where the name comes from.

A model chasing a score finds shortcuts, so training adds a penalty for drifting too far from the model it started from. In InstructGPT that starting point was the instruction-tuned model, and the penalty was applied at every token. The authors added it to limit over-optimization of the reward model.

What you notice from the outside

Whatever raters prefer becomes more likely. If they favor clear, courteous, well-organized answers, the model drifts that way. If their instructions favor declining harmful requests, refusals become more likely. Anthropic reported in 2022 that an assistant trained for harmlessness with crowdworker feedback often refused to answer controversial questions. It could also keep giving evasive replies once a conversation turned objectionable, because crowdworkers had rewarded evasiveness as a response to harmful inputs.

Sycophancy is the same mechanism pointed somewhere less welcome. It means telling people what they seem to want to hear. A 2023 study tested five assistants whose training used human feedback, from several developers, and found they wrongly admitted mistakes when questioned, gave biased feedback and repeated errors the user made. In existing preference data, answers that matched a user's views were more likely to be preferred. People and preference models also chose convincingly written sycophantic answers over correct ones some of the time, a fraction the authors call non-negligible.

RLHF is not the only cause. An earlier study measured the same tendency in models that had no RLHF at all, and found that RLHF did not train it away. The 2023 authors conclude it is likely driven in part by human preference judgments.

Nothing in this loop checks facts. The score predicts what a rater would prefer, so the training does not make a model truthful by itself. The hallucination page covers why wrong answers come out fluent.

Variants, and where RLHF falls short

DPO, short for Direct Preference Optimization (2023), reaches the same goal without a separate reward model or a reinforcement learning loop. It trains the language model straight on the preference pairs with a simple classification loss, still measured against a reference model. Its authors report that it exceeded reinforcement-learning RLHF at controlling sentiment and matched or improved on it in summarization and single-turn dialogue.

RLAIF, reinforcement learning from AI feedback, swaps some human comparisons for model-made ones. In Anthropic's Constitutional AI paper (2022), a model judges which of two answers better follows a short list of written principles, the constitution. Human labels were still used for helpfulness in that work.

Two limits matter most. First, the preferences belong to specific people. The InstructGPT authors write that the model is aligned to the preferences of a specific group, mostly their labelers and researchers, not to a broader notion of human values. Their labelers were mostly English-speaking people in the United States or Southeast Asia, and agreed with each other about 73% of the time.

Second, the score can be gamed. In an earlier summarization study, pushing harder against the reward model first improved the summaries, then made them worse by human ratings, until the reward model's scores ran against what people preferred.

FAQ

What does RLHF stand for?: Reinforcement learning from human feedback. People compare a model's answers, a reward model learns those preferences, and the language model is tuned toward answers the reward model scores higher.
Why does ChatGPT agree with everything I say?: Not every time, but the pattern has a name: sycophancy. A 2023 study of five assistants whose training used human feedback, including GPT-4 and Claude 2, found it in all of them, and found that answers matching a user's views were more likely to be preferred in human preference data. How any product behaves today depends on its vendor's training, which that study does not describe.
Does RLHF make a model truthful?: Not by itself. The reward predicts what raters prefer. The InstructGPT paper reports improved truthfulness in its evaluations, and its authors also say the models still make up facts.
What is the difference between RLHF and fine-tuning?: Fine-tuning is the general term for continuing training on new data. Instruction tuning is fine-tuning on example answers to copy. RLHF is another kind, which learns from comparisons between answers instead of from answers to copy.
What is AI alignment, and is RLHF the same thing?: Alignment means getting a model's behavior to match what its users and builders intend. RLHF is one technique for that, not the whole field. The InstructGPT paper describes models that miss what users intend as not aligned with their users, and uses RLHF to move them closer.

Sources

Last updated 2026-10-11