How much data do I need to fine-tune?

It turns on the task and the method. LoRA and other parameter-efficient methods can give useful results from a few hundred to a few thousand examples on a narrow task. Full fine-tuning for broader behavior change needs a lot more.

Does fine-tuning make a model worse at other things?

It can. Push too hard on a narrow dataset and the model gets worse at tasks outside it, an effect called catastrophic forgetting. It is one reason parameter-efficient methods, which touch fewer weights, are often the safer choice.

TRAINING TECHNIQUE

What Is Fine-Tuning a Model?

Fine-tuning takes a pretrained model and keeps training it on a smaller, more specific dataset, updating its weights so its behavior leans toward that new data. It is how a general base model becomes specialized for a task, a tone, or a domain.

What happens during fine-tuning

A pretrained model already has weights set by large-scale training. Fine-tuning runs the same process, gradient descent on a loss function, on a new and usually much smaller task-specific dataset. It does not wipe the model's existing knowledge. Its behavior shifts toward whatever patterns dominate the new data.

Full fine-tuning updates every weight. Parameter-efficient methods like LoRA update a small added set and freeze the rest, giving up a little of the quality ceiling for a large cut in compute and memory.

Fine-tuning vs. prompting vs. RAG

Three different tools, routinely mixed up. Prompting changes what you ask at inference time. It is fast to iterate and needs no training, but it does not change what the model knows or how it acts by default. Fine-tuning changes the weights, so the new behavior sticks without being restated in every prompt, but it costs real compute and a real dataset and does not reliably teach facts. RAG (see what RAG is) feeds in fresh, specific information at inference time and never touches the model.

The common mistake is fine-tuning in the hope the model will 'learn' new facts. Fine-tuning is good at format, tone, and task behavior. For new facts, reach for RAG.

When fine-tuning is the right call

Fine-tuning earns its cost when you need behavior a prompt cannot hold steady: a fixed output format at high reliability, a narrow domain vocabulary, or stripping out a default the base model keeps reaching for. If a well-written prompt or a retrieval step already does the job, fine-tuning is the slower and stiffer option.

FAQ

How much data do I need to fine-tune?: It turns on the task and the method. LoRA and other parameter-efficient methods can give useful results from a few hundred to a few thousand examples on a narrow task. Full fine-tuning for broader behavior change needs a lot more.
Does fine-tuning make a model worse at other things?: It can. Push too hard on a narrow dataset and the model gets worse at tasks outside it, an effect called catastrophic forgetting. It is one reason parameter-efficient methods, which touch fewer weights, are often the safer choice.

Sources

Howard & Ruder, 'Universal Language Model Fine-tuning for Text Classification' (2018)

Last updated 2026-09-06

What Is Fine-Tuning a Model?

What happens during fine-tuning

Fine-tuning vs. prompting vs. RAG

When fine-tuning is the right call

FAQ

Sources

Related