How LLMs learn

2026-02-15

A ground-up explanation of how language models are trained after pretraining. It sorts the techniques by where their training signal comes from, and follows them into the published DeepSeek-R1 pipeline, which interleaves them. It closes with the open question of whether RL teaches models anything their base could not already do.

Credit26 · mar 11Read notebanner image is a loss landscape visualization by Javier Ideami (losslandscape.com). I saw him present at an executive AI master's program and his conviction about where language modeling was heading stuck with me more than most of the curriculum did. his work on visualizing optimization surfaces is worth a deep dive if you want to see what gradient descent actually looks like.

Revisions26 · oct 05Read notecorrected several things the first version got wrong. DeepSeek's R1-Zero was told to reason inside think tags (what emerged was how long and how well it reasoned), and the R1 pipeline starts with a small SFT, not with RL. the 3 to 7% DPO gap measures how well its implicit reward ranks answers, not task performance. LoRA is no longer attention-only by default. and the claim that RL finds what the base model could never produce is now contested, so the article says so.

This is a companion piece to Transformers and local inference. That article covers the architecture, down to where the 175 billion parameters of GPT-3 live and what it costs to run them. This one covers what happens after the architecture exists and has read most of the internet: how a model that predicts the next token becomes one that follows instructions and declines to help you build a bomb.

The one-paragraph version of the other article: a transformer alternates attention blocks, which move information between tokens, with MLP blocks, which work on each token alone and hold most of what the model knows. It takes text in, produces a probability distribution over the next token, and writes by sampling from that distribution one token at a time. Every weight in it was set by backpropagation, adjusting the weights to reduce a loss function.

That last sentence is the bridge. All training, pretraining included, uses the same physical mechanism: backpropagation adjusts weights, every time. What separates the techniques is where the signal comes from that tells backpropagation which way to push.

Before we start

Four ideas carry everything that follows. If you know them, skip to the taxonomy.

BackpropagationThe loss functionPolicyInference and training

The transformer article described the forward pass, data flowing through the network to a prediction. Backpropagation is the trip back.

After the model predicts, you compare the prediction with the right answer and compute a loss, one number that says how wrong it was. Then you work backwards through the network and compute, for every one of the billions of parameters, whether nudging it up would raise the loss or lower it, and by how much. That is the gradient.

Then every parameter takes a small step in the direction that lowers the loss. One step barely changes anything; hundreds of thousands of them, over trillions of tokens, shape the model into something that predicts well. Nothing directs the process beyond that pressure to be slightly less wrong than before.

Every technique in this article uses it; what changes is how the loss is computed.

The loss is the number the model is trained to make small, which makes it the most consequential design choice in training: it defines what better means.

PRETRAININGHow surprised the model was by the real next token, measured as the negative log of the probability it gave that token.

SFTThe same quantity, computed on curated examples instead of raw internet text.

DPOWhether the model gives more probability to the preferred response than to the rejected one, relative to a frozen copy of itself.

RLA reward for the response the model just generated, turned into a loss by the policy-gradient machinery.

For pretraining and SFT that loss is cross-entropy, averaged over the tokens of a sequence:

L=−1T∑t=1Tlog⁡ pθ(xt∣x<t)\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T} \log \, p_\theta(x_t \mid x_{<t})L=−T1t=1∑Tlogpθ(xt∣x<t)

Each term asks what probability the model gave to the token that actually came next. Confident and right costs −log⁡0.9≈0.1-\log 0.9 \approx 0.1−log0.9≈0.1; clueless costs −log⁡0.01≈4.6-\log 0.01 \approx 4.6−log0.01≈4.6. Averaged over the sequence, that is the number gradient descent pushes down.

Reinforcement learning talks about the policy constantly; PPO is Proximal Policy Optimization. For a language model the policy is the model itself, the function that takes a context and returns a probability distribution over next tokens. Updating the policy means adjusting the weights so those probabilities change.

Every technique in this article updates the policy. They differ in the signal that says which way.

These are two moments in the life of a model, and the article moves between them.

Training is when the weights change: gradients flow, on clusters of thousands of GPUs, for weeks or months. Inference is when the model is used with its weights frozen: input in, output out. The KV cache from the transformer article is an inference mechanism and plays no part in training. Chatting with an assistant is inference; a lab spending tens or hundreds of millions of dollars on a new model is training.

RL blurs the line a little, because the model has to run inference to produce the responses it is trained on. Every technique here is still a training technique.

The taxonomy

Every way of training a model after pretraining falls into one of two families, set by whether the data exists before training starts:

Supervised learningthe data is fixed before training SFTimitate this example

DPOprefer response A over response B

Reinforcement learningthe model produces its own data RL (PPO, GRPO and others)maximise this reward

SFT and DPO are relatives. Both learn from a static dataset, built by people or by other models before the run begins, and both are trained to reproduce patterns in it.

RL runs a loop. The model is scored on what it generates and generates again with the weights that score changed, so the data it learns from is its own output, and can contain answers no dataset held.

The most common confusion in the field is filing DPO under RL, because it comes after SFT and makes the model better behaved. It has no loop, so it never explores. DPO is supervised learning with a cleverer loss. The three techniques follow in order.

SFT

Supervised fine-tuning is the simplest technique and usually the first step after pretraining.

You have a dataset of pairs, an input and the output you want, and you train the model to reproduce the outputs. The loss is the same cross-entropy as pretraining, measured only on the tokens of the desired output; what changed is the data, curated instead of scraped.

A model fresh from pretraining is good at continuing text and bad at being an assistant. Ask it What is the capital of France? and it may continue with This question is commonly found on geography quizzes and standardized tests, because on the internet a question is as often the start of a quiz as a request. It knows Paris; it has no reason to think you are asking. OpenAI's InstructGPT paper made the gap concrete: people preferred the answers of a 1.3-billion-parameter model fine-tuned to follow instructions over those of the 175-billion-parameter GPT-3 it started from1.

1Ouyang et al., Training language models to follow instructions with human feedback, 2022.

SFT fixes this with thousands of examples:

InputWhat is the capital of France?

OutputThe capital of France is Paris.

InputWrite a haiku about rain.

OutputSilver drops descend Dancing on the quiet earth Petals bow in thanks

InputExplain photosynthesis to a five-year-old.

OutputPlants eat sunlight! They use it to turn air and water into food...

The model learns the pattern: a question gets an answer, a request for a poem gets a poem. The facts were already there from pretraining, spread through the MLP weights as the transformer article describes; what SFT teaches is a format, how to respond to an instruction instead of continuing it.

What SFT changes in the model

Full SFT updates every weight, attention and MLP alike, and the changes are small compared with what pretraining did. Most fine-tuning in practice is cheaper than that. LoRA freezes the weights and trains a small low-rank correction beside some of the matrices, often well under one percent of the parameters, and matched full fine-tuning of GPT-3 175B in the paper that introduced it1. That paper adapted only the attention matrices WQW_QWQ and WVW_VWV, and attention-only LoRA became a habit, with a story attached about behaviour living in attention and facts in the MLPs. The QLoRA authors then found that matching full fine-tuning needed adapters on every linear layer of the block, MLPs included2, which is the usual default now. Where a fine-tuning change ends up living is an empirical question, and the clean split between format and facts is a simplification.

1Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021. 2Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023.

The tokenizer is never touched: changing it would orphan the embedding matrix and amount to starting over.

The whole model and the part SFT with LoRA rewrites today: every linear layer of the block, attention and MLP alike (the 2021 paper adapted only two attention matrices). The token embeddings and positional encodings stay frozen.

SFT and thinking

To make a model reason step by step, SFT is how you teach it the format. You write training examples like this:

userWhat is 47 × 83?

think<thinking>I need to multiply 47 by 83.47 × 80 = 3,76047 × 3 = 1413,760 + 141 = 3,901</thinking>

modelThe answer is 3,901.

masked, no lossreasoning, trained onanswer, trained onsplit at spaces for the picture; real tokenizers cut finer

The model learns that the opening <thinking> tag is followed by intermediate steps, then a closing tag and an answer, the same way it learns everything else, by predicting tokens. SFT teaches what reasoning looks like, and only as well as the examples reason. The model fills the block because statistically that is what follows (causal language modeling), whether or not the steps help the answer1. It imitates the shape of good reasoning, and it can reach reasoning better than its own only if the examples carry it. Distillation is that case: DeepSeek fine-tuned small Qwen and Llama models on 800,000 samples written by its large reasoning model, and the small models reasoned better than the same small models trained with RL directly2.

1Why intermediate tokens help at all has a precise answer. Each generated token is another pass through the same layers with a longer input, so a chain of thought composes the network with itself, f=fn∘⋯∘f1f = f_n \circ \cdots \circ f_1f=fn∘⋯∘f1, with each fif_ifi one decoding step. Merrill and Sabharwal showed that a transformer allowed a polynomial number of such steps can solve exactly the problems solvable in polynomial time, and with a linear number at least all regular languages, both beyond what it can do in one pass (The Expressive Power of Transformers with Chain of Thought, ICLR 2024). That is the ceiling on what extended thinking can express; which steps are worth taking is what training has to teach. 2DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025, section 4.1.

SFT answers the question of what goes here. The question of what is worth putting here, when nobody has written a better example, needs RL.

DPO

Direct Preference Optimization goes one step beyond SFT. Instead of one response to imitate, each example is a pair, a better and a worse response to the same prompt, and the model learns from the contrast:

PromptExplain quantum computing.

ChosenQuantum computing uses qubits that can be in superposition...

RejectedWell, quantum computing is very complicated and hard to explain...

The model is trained to raise the probability of the chosen response and lower that of the rejected one, measured against a frozen copy of itself:

LDPO=−E[log⁡σ ⁣(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}} = -\mathbb{E} \left[ \log \htmlData{sym=4}{\sigma} \!\left( \htmlData{sym=5}{\beta} \log \frac{\htmlData{sym=2}{\pi_\theta}(\htmlData{sym=0}{y_w} \mid x)}{\htmlData{sym=3}{\pi_{\text{ref}}}(\htmlData{sym=0}{y_w} \mid x)} - \htmlData{sym=5}{\beta} \log \frac{\htmlData{sym=2}{\pi_\theta}(\htmlData{sym=1}{y_l} \mid x)}{\htmlData{sym=3}{\pi_{\text{ref}}}(\htmlData{sym=1}{y_l} \mid x)} \right) \right]LDPO=−E[logσ(βlogπref(yw∣x)πθ(yw∣x)−βlogπref(yl∣x)πθ(yl∣x))]

ywy_wywthe preferred response yly_lylthe rejected one πθ\pi_\thetaπθthe model being trained πref\pi_{\text{ref}}πrefthe frozen reference, usually the SFT checkpoint σ\sigmaσa sigmoid, squashing the gap between the two into a probability β\betaβhow sharply the model should prefer the winner

The ratios πθ/πref\pi_\theta / \pi_{\text{ref}}πθ/πref are the anchor: the loss rewards moving probability toward the winner relative to where the reference had it, and the further the model drifts from the reference, the less it gains. Rafailov and colleagues derived the loss from the RLHF objective itself, reward plus KL penalty, and showed it can be optimised as a classification loss on preference pairs with no separate reward model and no RL loop1. The preference data is the same as RLHF's, and the training run behaves like ordinary fine-tuning.

1Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023.

Why DPO is not RL

DPO feels like RL because it refines behaviour after SFT, in the alignment stage of the pipeline. It differs in the one respect that defines RL. Think of flash cards and a match. Flash cards show you a right answer and a wrong one and you learn to tell them apart. A match makes you perform and tells you how it went, so next time you can try something else. Both make you better, and only the match can teach you a move that was never on a card.

DPO

pairs of better and worse responses, written beforehand The data the model's own responses, scored as they come

a static dataset, processed in epochs How it runs a live loop

no Generates during training yes, and updates between rounds

In both Weights move by: backpropagation

DPO never generates during training. The comparisons exist before the run starts, and the model learns from them exactly as SFT learns from fixed examples. So DPO can only prefer among behaviours that appear in its pairs: if neither response in any pair uses a good strategy, the model learns the lesser of two evils and never meets the good one.

Where DPO falls short

Lin and colleagues measured one of DPO's weak spots directly. DPO implicitly learns a reward (the log-ratio in its loss), and on data like its training data that implicit reward ranks responses about as well as an explicit reward model trained on the same preferences; on five out-of-distribution test sets it ranked them 3% less accurately on average and 7% less in the worst case1. That is a gap in judging answers, which is upstream of task performance and a reasonable explanation for why an explicit reward model transfers better to unfamiliar prompts.

1Lin et al., On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization, EMNLP Findings 2024.

For subjective qualities such as tone or safety DPO works well, because people's preferences are exactly what the pairs record. For reasoning it is limited. You can rank two chains of thought, but the model learns to prefer among the strategies in the data and finds none outside it.

Reinforcement learning has a loop, and that is the whole difference:

1The model answers a prompt, sampling a response from the weights it has right now2A reward function scores that response with a single number, a verdict without a corrected answer3Backpropagation shifts the weights toward whatever earned the higher score4The updated model answers again, and those new responses are the next round of training data

The weights change during the process, and each round of generation comes from the weights the round before produced. That makes RL online, with training data produced by the model itself as it improves, and it means the model can be pushed toward answers that no example showed it. How far that goes beyond what the model could already do is the most argued question in this article, and the last section of RL comes back to it.

The DeepSeek-R1-Zero experiment

The clearest demonstration is DeepSeek's R1-Zero. The researchers took a pretrained model, DeepSeek-V3-Base, with no SFT and no human-written examples of reasoning, and ran RL with two rule-based rewards: one for a correct final answer on maths and code problems that can be checked automatically, and one for format, putting the reasoning between think tags and the answer after them1.

1DeepSeek-AI, DeepSeek-R1, 2025, section 2.2. The training template told the model to think about the reasoning process first and then answer, inside <think> and <answer> tags.

So the model was told where to reason. Nobody showed it how, or how long. Over training its responses grew from hundreds of tokens to thousands, and it began to reread its own steps and start again when it found a mistake (the paper reproduces one such moment, where the model writes that it has had an aha moment and reconsiders). All of it came from the reward for correct answers and nothing else, which is why chain-of-thought reasoning of that quality emerging from RL alone surprised people.

It was also messy. R1-Zero's reasoning was hard to read and switched between languages in the middle of a thought, English and Chinese mixed like a mathematician scribbling in whichever comes first. A model that works but cannot be read is not a product, which is why the released R1 used a longer recipe.

The DeepSeek-R1 pipeline

The published pipeline for R1 shows how the techniques combine1:

1DeepSeek-AI, DeepSeek-R1, 2025, section 2.3.

COLD STARTA small SFT on thousands of long, readable reasoning examples (some written by prompting, some cleaned from R1-Zero's outputs by people), so RL starts from a model that already reasons in a legible format.

REASONING RLLarge-scale RL with GRPO on checkable maths, code and science problems, adding a reward for keeping the reasoning in one language.

REJECTION SAMPLING AND SFTThe RL checkpoint generates many answers per prompt, only the correct and readable ones are kept (about 600,000 reasoning samples, plus 200,000 general ones), and the base model is fine-tuned afresh on them.

RL AGAINA second RL stage across all kinds of prompts, with rule-based rewards for reasoning and reward models for helpfulness and harmlessness.

Read in terms of the range of the model's behaviour, each RL stage widens it. Rejection sampling narrows it back to what is useful, and the SFT on the kept samples fixes the result in fresh weights for the next RL stage to push from.

holdCold-start SFTthousands of readable reasoning examples set the format expandReasoning RL with GRPOcorrectness and language-consistency rewards contractRejection sampling and SFTkeep what is correct and readable, retrain the base expandRL across all promptsreasoning rewards plus helpfulness and harmlessness

Illustrative: width is the variety of behaviors, height is how useful they are

RL and rejection sampling get confused, and the difference is when the weights move. In RL the model updates between rounds of generation, so later responses come from a model the earlier ones already changed. Rejection sampling is offline: the weights stay frozen while a large batch is generated and scored, and only the best of it is trained on (with SFT or DPO) in a separate phase. Rejection sampling can only select among behaviours the model already produces, and RL can move the model toward ones it produced rarely enough to be worth nothing to a filter.

What RL adds, and how much

The strong version of the story is that RL finds behaviours no dataset contained and the base model could never produce. R1-Zero is the usual evidence, and it is weaker than it looks. Yue and colleagues sampled base models and their RL-trained versions many times per problem and found that the RL models win when each gets one or a few attempts, while the base models solve more distinct problems when both get hundreds; their reading is that RL with verifiable rewards concentrates probability on reasoning the base model could already produce1. An NVIDIA team answered that with much longer RL training the trained models solved problems the base model failed at under any amount of sampling2. Both can be true at once: RL makes the right answer the likely one, which is most of what a user experiences, and with enough of it, and enough of a gap in the base, it may also reach answers the base never gives.

1Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, NeurIPS 2025. 2Liu et al., ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models, 2025.

A cybernetic reading: requisite variety.

W. Ross Ashby's law of requisite variety says a regulator can only hold an outcome steady if it has at least as many distinct responses as the disturbances it faces; in Ashby's words, only variety can destroy variety1. Read through it, SFT and DPO map fixed inputs to fixed outputs, with a variety bounded by what the training data shows, while RL is a feedback loop whose variety can grow because the loop generates behaviours and tests them. The analogy has a limit worth stating: SFT can raise a model's variety when its data comes from a system that has more (the distillation result above), so the bound is the data's variety, not SFT's. It is a lens on why RL suits problems with no good demonstrations, and it settles nothing about the debate in this section.

1Ashby, An Introduction to Cybernetics, 1956, chapter 11. The more quoted variety absorbs variety is Stafford Beer's restatement.

RLHF

Reinforcement learning from human feedback is the best-known term in the field and a widely misread one: people hear human feedback and imagine someone scoring each response as the model trains. The human work happens before the loop, in two phases, after the SFT step it usually follows.

Phase 1: collect human preferences

People compare pairs of model responses to the same prompt and pick the better one. The result is a dataset of prompt, chosen response and rejected response, exactly the format DPO consumes. This is the only phase with people in it; once the dataset exists, they are done.

Phase 2: train a reward model, then run RL

A separate neural network, the reward model, is trained on those preferences to predict, for a prompt and a response, how much a person would like it, as a single number. Then RL (classically PPO, Proximal Policy Optimization) uses that reward model as the judge, moving the language model's weights toward the responses it scores higher. InstructGPT is the reference recipe: SFT, then a reward model, then PPO1.

1Ouyang et al., Training language models to follow instructions with human feedback, 2022.

The two phases of RLHF. On the left, the part that happens once: humans compare responses, and their choices train the reward model. On the right, the part that repeats for many iterations: the policy answers, the reward model scores, PPO updates the policy. No human appears in the loop.

The human feedback is historical and frozen, compressed into the reward model. During RL no person looks at anything; the reward model is a proxy, an automated judge that approximates what the annotators would have said.

Humans leave before the loop starts.

The name says the feedback came from humans; it does not put them in the training loop. Their judgements are captured once, compressed into a reward model, and the RL loop then runs on its own, often for days on thousands of GPUs. A single run can generate millions of responses, far more than any team of people could read, which is the point of the reward model.

The reward model on its own

Once trained, the reward model is a standalone system that can score any response, and it gets used outside RLHF, most often at inference to rank several candidate answers and keep the best (best-of-N sampling). Labs also use it to filter a dataset before training, or in production as a classifier. It outlives the training run that made it.

DPO and RLHF side by side

RLHF1Human preferences2Reward model3PPOan RL loop

DPO1Human preferences2Direct optimizationno reward model, no RL

Both start from the same preference triplets and run different algorithms on them. DPO is cheaper and simpler and trains like fine-tuning. RLHF costs more and is harder to tune, and buys a reusable reward model and a loop that keeps generating fresh data. The data format commits you to neither: the same pairs can feed RLHF or DPO.

RLAIF and Constitutional AI

Preference data is expensive. It takes thousands of hours of careful comparison by trained annotators, which raises the question of whether another model can do the judging.

RLHF1Humans compare responses2Reward model3PPO

RLAIF1A model compares responses2Reward model3PPO

That is RLAIF, reinforcement learning from AI feedback: a language model, prompted with a set of principles, picks the better of two responses, and its picks train the reward model. Google researchers found it reached results comparable to RLHF on summarisation and dialogue tasks1.

1Lee et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback, 2023.

Constitutional AI, from Anthropic, is the best-known version, and it has two phases1. In the supervised phase the model critiques its own answer against a written list of principles (the constitution) and is fine-tuned on its revisions. In the RL phase a model judges pairs of responses against the same principles, and RL runs against a preference model trained on those judgements. The human input is the constitution itself, plus the human preference data the paper still used for helpfulness.

1Bai et al., Constitutional AI: Harmlessness from AI Feedback, 2022.

This has the model grading its own homework in part, and the result is only as good as the principles and the judge applying them. A vague constitution or a weak judge produces garbage with a principled veneer.

KL divergence

All of these techniques share a danger, reward hacking. A model optimised against a number has no stake in what the number was meant to measure, and wherever the two come apart, optimisation finds the gap, the way water finds the crack in a dam.

A model trained on a helpfulness reward can learn to be verbose and agreeable (the coworker who answers every question with a three-paragraph email full of exclamation marks), because longer, more affirming answers tend to score higher with reward models even when a short honest one would serve better. A model trained to pass unit tests can learn to special-case the tests instead of solving the problem. You asked for passing tests and got them.

The standard defence is a penalty on KL divergence, a measure of how far one probability distribution is from another. During RL the loss charges the model for moving away from a reference, usually the SFT checkpoint:

Loss=−Reward+β⋅DKL(πθ∥πref)\text{Loss} = -\text{Reward} + \htmlData{sym=2}{\beta} \cdot \htmlData{sym=3}{D_{KL}}(\htmlData{sym=0}{\pi_\theta} \| \htmlData{sym=1}{\pi_{\text{ref}}})Loss=−Reward+β⋅DKL(πθ∥πref)

πθ\pi_\thetaπθthe model being trained πref\pi_{\text{ref}}πrefthe reference model, usually the SFT checkpoint β\betaβthe tradeoff: higher keeps the model closer to the reference, lower gives it more freedom to change DKLD_{KL}DKLhow far the two distributions are from each other

The KL term lets the model change, within limits.

The reference policy (blue) anchors the trained policy (violet). Optimization pulls it towards the region where the reward is hacked, and the KL penalty is the dashed circle it is not allowed to leave; β decides how wide that circle is.

With a learned reward model the anchor is close to mandatory: without it the policy drifts into degenerate text that the reward model happens to score highly and nobody wants. With rewards that can be checked exactly (a correct maths answer, passing hidden tests) the gap to exploit is smaller, and some recent recipes for training reasoning drop the KL term altogether, arguing that a reasoning model has to move far from where it started; ByteDance's DAPO is one1.

1Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025.

The objective has a closed-form optimum, π∗(y)∝πref(y) er(y)/β\pi^*(y) \propto \pi_{\text{ref}}(y), e^{r(y)/\beta}π∗(y)∝πref(y)er(y)/β, so the anchor can be moved by hand. Below, eight behaviours with a made-up reference probability and reward; lower β\betaβ and watch the probability run to the one that games the tests.

Answers directly20% Shows its steps22% Hedges7% Refuses2% Three-paragraph email17% Flatters the user16% Games the tests15% Off-topic0%

β = 1.5

DPO carries the same anchor inside its loss, since it is derived from this objective. That is part of why it trains stably: β\betaβ is still a knob, but there is no sampling loop and no separate reward model whose blind spots the policy could chase.

On-policy and off-policy

On-policy and off-policy data differ in who generated it. On-policy data comes from the current version of the model, and it goes stale with the first update, so you generate again. Off-policy data comes from anyone other than the current model, an older version of it included.

SFT and DPO train off-policy, on data written before the run. Rejection sampling is off-policy too once the kept samples are reused for training. PPO and GRPO are on-policy (in practice nearly so, reusing each fresh batch for a few updates and clipping how far one batch can move the model), which is why they are expensive: the model has to keep generating.

The trade-off runs both ways. On-policy data shows the model its own current mistakes, which gives the most relevant gradient but spends a full generation on every few updates. Off-policy data can be reused as often as you like and is cheaper per update, but the further its author is from the current model, the less it says about what this model does wrong.

Cursor's Tab model is about as on-policy as production gets. Cursor rolls out new checkpoints to users several times a day; each cycle, from deploying a checkpoint to having its data for the next update, takes an hour and a half to two hours1. When a user accepts a suggestion or ignores it, that signal is about the current model's behaviour with current users.

1Cursor's post on online RL for Tab, September 2025.

Safe RLHF

A single reward model scoring overall quality merges at least two things that conflict: helpfulness (does the answer help?) and harmlessness (does it avoid causing harm?). A maximally helpful answer to how do I pick a lock? is a detailed tutorial and a maximally cautious one is a refusal, and a single score has to settle that conflict somewhere inside the reward model, where nobody can see the setting.

Safe RLHF takes the conflict out into the open1. Annotators judge helpfulness and harmlessness separately, so a reward model can learn the first and a cost model the second. RL then maximises reward subject to a limit on cost, with a Lagrange multiplier that rises when the model gets less safe and relaxes when it is safe enough. The balance becomes a setting you can read and adjust, instead of something a single reward model learned implicitly.

1Dai et al., Safe RLHF: Safe Reinforcement Learning from Human Feedback, 2023.

Model collapse

Model collapse is the failure mode that haunts every pipeline built on synthetic data. When a model is trained on output generated by another model, or by an earlier version of itself, the output distribution loses diversity: the common patterns are reinforced and the rare ones, the tail of the distribution, thin out with each generation. Shumailov and colleagues showed it across model types, with the tails disappearing first1.

1Shumailov et al., AI models collapse when trained on recursively generated data, Nature 631, 2024.

After enough rounds the model converges on a narrow set of safe outputs, a photocopy of a photocopy that loses a little contrast at each pass until only a grey smear of the original is left. The tails go first, and they hold the edge cases, which is much of what makes a model useful on the problems nobody anticipated.

Each student learns from the outputs of the model before it. The secondary peaks go first, then the tails, and what is left after enough generations is one narrow mode.

Three of the techniques above lean on synthetic data, and each carries the risk in its own way:

RLAIFTrains on judgements generated by a model, so the judge's blind spots become the student's, especially when the two are close relatives.

REJECTION SAMPLINGFilters model output, and by construction keeps the common good answers and drops the unusual ones.

DISTILLATIONTrains a small model on a large one's output; the small model inherits the large one's modes and little of its spread.

The known antidote is to keep real data in the mix: Shumailov's own experiments found that preserving part of the original human data slowed the collapse. That is one reason labs keep paying for human annotation as generated data gets cheaper: the tail of the distribution is the part hardest to capture and the first to go1.

1Each training stage is a distributional shift. Pretraining sets the broad distribution, and each stage after it narrows the distribution toward what that stage rewards. Each shift trades breadth for usefulness, and collapse is what happens when the narrowing compounds over generations until only the mode is left.

The cost of thinking

A reasoning model that writes a long thinking block before answering generates many more tokens per response, and tokens are compute. A model that thinks for 2,000 tokens before a 200-token answer spends ten times the generation of one that answers directly; a hard maths problem can take 20,000 thinking tokens, fifty cents at $25 per million output tokens before a word of the answer appears. Providers bill thinking tokens as output tokens, the expensive kind, and let you cap how many a request may spend.

MORE THINKINGBetter answers on hard problems, higher cost and latency.

LESS THINKINGFaster, cheaper, sometimes worse on complex tasks.

NO THINKINGCheapest and fastest, and enough for simple factual questions.

SFT and RL together decide how a model spends that budget. SFT teaches the format of reasoning; RL rewards correct answers, and a model trained that way learns to spend more tokens where they change the outcome. In the R1-Zero run the response length grew by itself as the problems got solved more often, with no reward for length.

An information-theoretic reading of thinking.

Before the model answers, it faces a distribution over continuations, some right and most wrong, and that distribution has an entropy: high when many tokens are plausible, low when the model is confident. Reasoning tokens condition the answer on more context, each one a distributional shift toward the region where correct conclusions are likely, so the answer is sampled from a narrower distribution. The thinking adds no knowledge; it moves the model to the place in its own latent space where it already had the answer.

That suggests a ceiling, by analogy with channel capacity: thinking cannot carry the model past what its weights support, which is one reason scaling the base model keeps mattering. The pass-at-many-attempts results discussed above fit that reading, and the longer-RL results suggest the ceiling moves when the weights do. The flip side is overthinking: on easy problems, where the entropy is already low, extra reasoning can walk the model away from an answer it had right.

Cursor as a case study

Cursor, the code editor, runs two RL systems that make the ideas above concrete.

Cursor Tab: online RL from accepts and rejects

The autocomplete model predicts your next edit on every keystroke, over 400 million requests a day. It is trained with a policy-gradient method on what users do: a suggestion that is accepted earns a reward of 0.75, one that is rejected costs 0.25, and staying silent earns nothing1. The arithmetic makes a suggestion worth showing only when the model thinks it will be accepted more than a quarter of the time. There is no chain of thought and no reasoning, only a choice of whether to suggest and what.

1Cursor's post on online RL for Tab, September 2025.

New checkpoints reach users several times a day and the model retrains on what they did with the last one. Cursor reports that the new model makes 21% fewer suggestions and has a 28% higher acceptance rate on the ones it makes: it learned when to stay quiet.

Composer: RL with tools

Composer, Cursor's agent model, gets a problem description and tools (reading and editing files, a terminal, codebase search) and has to act on the problem, in hundreds of thousands of sandboxed coding environments at once. Cursor writes that during RL the model learned on its own to run complex searches. It also began fixing linter errors and writing and executing unit tests1.

1Cursor's post introducing Composer, October 2025.

Nobody wrote search the codebase before editing into a training example. Those are emergent behaviours: each habit correlated with higher rewards, so it was reinforced1. It is the same dynamic as R1-Zero lengthening its reasoning, in a different domain. SFT could teach search-before-edit if someone wrote examples of it. What RL adds is that nobody had to.

1This is second-order emergence. First-order emergence is one surprising behaviour, the model learning to search. Second-order is when separately emergent behaviours compose into a workflow nobody designed, such as searching before an edit and testing after it. The same holds in extended thinking, where single steps are first-order and the ability to sequence them into a proof is second-order.

Putting it all together

The modern pipeline is a sequence, each step answering a different need:

PRETRAININGThe model reads trillions of tokens of text and code and learns their patterns by predicting the next token. The result is a powerful text predictor with no particular manners. The architecture (attention moving context, MLPs holding most of what is known) is fixed from here on.

SFTThe model learns to follow instructions, and to reason inside thinking blocks when its examples do. Same loss, curated data. The result imitates good behaviour and is bounded by the quality of what it imitates.

RL, RLHF OR RLAIFThe model improves in a loop on scores for its own output, and optimises for being right or being preferred instead of for imitation. The loss is a reward, usually with a KL anchor.

DPO OR MORE SFTRefinement on specific preferences such as tone or safety, and consolidation of what RL found. It is supervised and stable.

Each answers a different question:

SFTWhat does a good response look like?

DPOWhich of these responses is better?

RLWhat is worth doing, even if no one has shown you?

They are stages more than competitors, and the most capable models use all of them, in sequence and sometimes more than once, because each does something the others cannot.

Reasoning models make the point sharpest. Nobody tells them what to think. They are rewarded for right answers, and from that signal alone they learn that spending tokens on intermediate steps helps, and some of what makes those steps good. The thinking is emergent, like the specialisation of attention heads in a transformer: nobody designs it, and it falls out of the optimisation, from a pile of random numbers that was only ever told to be less wrong.

The signal ladder

The summary above says what each technique answers. When to use which is a separate question, and the answer depends less on which is most powerful than on which failure you can live with. Each step up the ladder gives the model more freedom, to find good behaviours and bad ones, and the kind of thing that goes wrong changes with it.

SFT: you get what you showed it

The signal is examples and the model imitates them. The failure is a ceiling: the model does not exceed the quality of its examples. If the annotators wrote mediocre reasoning, the model produces mediocre reasoning, confidently and in perfect format, and it inherits their blind spots. In exchange it is predictable. It stays close to the distribution you showed it, and its worst case is bounded by your worst examples, which you control.

DPO: it can rank, but it cannot invent

The signal is preferences and the model learns to tell better from worse. The failure is coverage: it prefers among behaviours that appear in the pairs, so if no pair contains a good strategy, it learns the less bad one and never meets the good one. Its implicit reward also transfers less well to unfamiliar prompts than an explicit reward model, by the few points measured above. In exchange it is cheap and trains like fine-tuning.

RL: it will find every shortcut

The signal is reward and the model explores. The failure is reward hacking: every gap between the reward and what you want will be found and used. In exchange it is the only technique here that can move a model toward behaviours nobody demonstrated, R1-Zero's long self-checking reasoning and Composer's search before editing among them, with how far beyond the base model that reaches still argued. Its ceiling is the quality of the reward (or of the reward model that stands in for it), and keeping the run honest takes a KL anchor or exact rewards, with someone watching.

So which one do you use?

Technique Freedom Failure mode Ceiling

SFT low imitation data quality

DPO medium coverage gap comparison diversity

RL high reward hacking reward quality

The useful question is which failure mode you can afford.

INSTRUCTION FOLLOWINGSFT. Its failure (it cannot exceed the data) is acceptable because you control the data.

STYLE OR SAFETYDPO. Its failure (it invents nothing new) is acceptable because you want the model close to demonstrated human preferences.

REASONING AND TOOL USERL. Its failure (reward hacking) is the price of a model that gets better than its examples, and checkable rewards lower that price.

Most production pipelines use all three because each covers a failure the others cannot. At each step the question worth asking is whether you can handle what goes wrong when you use it.