ML//model//GPT//nanoGPT

nanoGPT is Andrej Karpathy's minimal pre-training codebase for reproducing GPT-2 from scratch: about 600 lines of core training code, a research instrument rather than a product and the reference implementation for understanding the pre-training loop (dataset, forward pass, loss, backprop, repeated). GPT-2 at 124M is the standard benchmark for research on optimizing pre-training.

nanoGPT is Andrej Karpathy's minimal pre-training codebase for reproducing GPT-2 from scratch: about 600 lines of core training code, a research instrument rather than a product and the reference implementation for understanding the pre-training loop (dataset, forward pass, loss, backprop, repeated). GPT-2 at 124M is the standard benchmark for research on optimizing pre-training.

Its successor nanochat (2025) followed nanoGPT (2023) and added fp8 mixed precision, better data loading and fused kernels. It trains GPT-2 (d12, 124M parameters) on a single 8×H100 node in about two hours, and that speed enables rapid ablation, 8 to 12 experiments per day.

Dataset quality dominates: NVIDIA ClimbMix beat FineWeb-edu, DCLM and OLMo data out of the box. That raises Goodhart's curse concerns, the same risk as in training a reward model: ClimbMix may be optimized for the benchmarks, and a model trained on it may score better without being better.

Karpathy tested AI agents (Claude Code, Codex) as automated researchers iterating on nanochat. The agents implement well-scoped ideas perfectly but can't design experiments: no ablation discipline, no baseline control, spurious findings (e.g. they discovered that bigger networks have lower loss without controlling for FLOPs).