ML//training//pre-training

2026-02-15

Pre-training is pure next-token prediction at massive scale, with no human feedback and no instructions, and it sets what a model can ever do: it gives the model language, and SFT later gives it obedience.

The objective fits in one line: over trillions of tokens of web text, books and code (much of it filtered from Common Crawl), the model learns to assign high probability to each actual next token given the ones before it, and the cross-entropy of that prediction is the loss.

P(xt∣x1,…,xt−1)P(x_t \mid x_1,\ldots,x_{t-1})P(xt∣x1,…,xt−1)

Predicting the next token well forces the model to absorb syntax, facts, code structure and the relations between concepts, because all of them make the next token less surprising. It is the most expensive stage by far (training cost), and everything after it is post-training.

A baby alien staring at the static of the universe.

It stares at billions of sentences until it realizes that the pilot has turned on the... is followed by seatbelt sign and not by karaoke machine.

Pretraining is the ceiling. SFT and DPO or RL create no new capabilities; they navigate toward useful regions of the distribution already learned here. If the pretraining data lacks high-quality reasoning, no amount of RL recovers it, and a small model with much RL stays below a large model with little RL.

This makes model collapse especially dangerous: the sophisticated reasoning that lives in the tail distribution is the first thing erased by recursive synthetic training, and once gone from pretraining, extended thinking can't reach those regions.

Continued pre-training resumes the same objective on a mountain of niche text, which is like sending a toddler who just learned to speak to medical school.

In the GPT family, GPT-2 was pure pre-training (no fine-tuning). OpenAI wanted to see if volume alone was enough.

GPT-3 was the same but massive. It predicted text beautifully but didn't follow instructions.