Predicted and measured costs of training a character-level transformer on a CPU

A 4.75-million-parameter transformer is trained from scratch with nanoGPT on 1,976 posts by a Spanish poet, on the CPU of a laptop. The corpus is measured first and turned into one integer per character, and the run's starting loss, length, price and final loss are predicted from rules of thumb and a ten-step benchmark before anything is trained. The predictions are then checked against the run. The one that failed did so because dropout inside attention pushes PyTorch's CPU attention onto a kernel eleven times slower.

Karpathy's lecture Let's build GPT: from scratch, in code, spelled out spends its first quarter of an hour on what looks like bookkeeping. He opens a text file, prints how long it is, lists the characters it contains and gives each one an integer. I was halfway through that part when I paused the video and decided to run the whole thing myself on text I care about, which in my case is one of my favourite poets: a Spanish writer whose public posts I had scraped into a folder months ago for a local experiment, 1,976 of them. The code is nanoGPT, Karpathy's minimal GPT training repository, cloned as it is except for one line I come to later. The machine is the ThinkPad of the integrated GPU piece, a Ryzen AI 7 PRO 450 with 64 GB of RAM, and I installed PyTorch's CPU build, so every number below comes from its CPU (the integrated Radeon sat this one out).

I set myself one rule, the way people who train large models are said to work: write every number down before running anything. How big the data is, how big the vocabulary, where the loss starts, how long the run takes, what it costs and where the loss should end. Only then train, and see which predictions survive. The corpus and the trained weights stay on the laptop; the samples at the end are generated text, a few lines long.

Measuring the corpus before touching the model

So the first thing is to look at the data. Each post is a plain text file of about two thousand characters, captions written to go under a photograph: short paragraphs, a lot of first and second person, and the social residue around them (the photographer's handle, a venue, a hashtag).

Raw corpus posts1,976 characters3,629,723 words647,675 median post2,079characters distinct characters205 @ mentions273 hashtags = 175

For scale, the tiny Shakespeare file Karpathy uses has 1,115,394 characters and 65 distinct ones1, so this is a bit more than three Shakespeares with three times the alphabet. The alphabet is bigger because Spanish brings accents, ñ, ¿ and ¡, and because people writing on a phone bring emoji.

1nanoGPT's shakespeare_char readme gives 1,003,854 training and 111,540 validation tokens, one token per character.

Two things go before any of it reaches the model. The mentions and hashtags go first: they are names of other accounts, and a model trained on them would learn to invent plausible handles, which is the one thing nobody wants from it. Then the rare characters. After the cleanup there are 202 distinct characters, and the count of each one falls off a cliff.

Every character that appears in the cleaned corpus, most frequent first. The 80 to the right of the cut appear fewer than five times each: a few Arabic letters from one post, rare emoji and symbols.

Every character in the vocabulary gets a row in the embedding matrix and a column in the output layer, and that row only learns when the character shows up in a training example. A character seen twice in 3.6 million gets two gradient updates over the whole run, its row stays close to its random starting point, and at sampling time it would come out by accident or not at all. So I dropped every character seen fewer than five times. That removes 80 symbols and leaves a vocabulary of 122 characters, the number the rest of the run is built on.

One integer per character

The model never sees text. It sees integers, and something has to decide which integers. Karpathy's choice in the lecture is the simplest one there is: sort the distinct characters and number them.

PYTHON Copy

chars = sorted(set(text)) # the 122 distinct characters stoi = {ch: i for i, ch in enumerate(chars)} # character to integer itos = {i: ch for i, ch in enumerate(chars)} # integer to character encode = lambda s: [stoi[c] for c in s] decode = lambda l: "".join(itos[i] for i in l)

encode("hola") # [64, 71, 68, 57]

Then the whole training text goes through encode once and is saved as a flat array of 16-bit integers, train.bin, which is the same thing as the torch.tensor(encode(text)) of the video written to disk. That is the entire tokenizer.

It is also the least efficient one: one token per character means the model's 256-position window holds about 45 words. The two alternatives worth measuring are the ones the video and the GPT world actually use. One is GPT-2's BPE vocabulary of 50,257 tokens through OpenAI's tiktoken (tiktoken.get_encoding("gpt2")). The other is a SentencePiece BPE model trained on this text, the kind of tokenizer Llama 2 uses. I trained SentencePiece on the training split at three vocabulary sizes and measured all of them on the validation split.

Compression against size. A SentencePiece vocabulary of 1,000 tokens learned on these posts already packs more characters per token than GPT-2's 50,257, whose merges were learned mostly on English web text. At the model's width of 256, GPT-2's embedding table alone would be 2.7 times the whole model.

GPT-2's tokenizer is the bad deal here. Its merges were learned on English, so Spanish words get cut into short pieces (2.64 characters per token, against the four or so it manages on English), and its table of 50,257 rows would cost 12.9 million parameters at width 256, most of them rows for English subwords that never occur in 3.6 million characters of Spanish. It makes sense only to fine-tune GPT-2's own weights, which come with it.

SentencePiece is the real alternative, and a vocabulary of 4,000 would let the same window see almost four times as much text. I kept characters for this first run anyway, for two reasons. It is what the lecture does, so every number can be checked against Karpathy's. And with 3.6 million characters of data, a 4,000-token vocabulary would turn the corpus into about 950,000 tokens spread over 4,000 rows, roughly 240 examples per row on average and far fewer for the rare ones, against about 30,000 per character.

The two runs could not be compared on their losses directly, because the loss is measured per token and a token means something different in each. The fair unit is bits per character: divide the loss in nats per token by the characters per token and by ln⁡2\ln 2ln2.

bits per character=Ltokenc⋅ln⁡2\text{bits per \htmlData{sym=1}{c}hara\htmlData{sym=1}{c}ter} = \frac{\htmlData{sym=0}{\mathcal{L}_{\text{token}}}}{\htmlData{sym=1}{c} \cdot \ln 2}bits per character=c⋅ln2Ltoken​​

Ltoken\mathcal{L}_{\text{token}}Ltoken​cross-entropy per token, in nats ccccharacters per token on the same text

For the character model c=1c = 1c=1, so its validation loss divided by ln⁡2\ln 2ln2 is already the number to beat.

Random chunks of 256 characters

Karpathy splits his file at 90%: the first nine tenths are training, the last tenth validation. I held out 10% too, but of whole posts chosen at random with a fixed seed, 197 of them and 370,539 characters. A cut at a position would split one post across the two sets, and the files are numbered in what is probably publication order, so the last tenth would be a different stretch of the writer's years. A random sample of whole posts avoids both.

Training never reads the text in order. Each step, nanoGPT picks random offsets into train.bin and cuts a chunk at each one:

PYTHON Copy

ix = torch.randint(len(data) - block_size, (batch_size,)) x = torch.stack([torch.from_numpy((data[i:i+block_size]).astype(np.int64)) for i in ix]) y = torch.stack([torch.from_numpy((data[i+1:i+1+block_size]).astype(np.int64)) for i in ix])

block_size is the longest context the model ever sees, and y is x shifted by one. That shift is the whole trick of the lecture: a chunk of nine characters is eight training examples at once, because the model is asked for the next character after every prefix of it.

The first nine characters of the training split with block_size = 8, the size Karpathy uses to explain it. Every position predicts the character after it, seeing only what came before.

In the real run the chunk is 257 characters, so each one holds 256 examples, and a batch of 32 chunks asks for 8,192 predictions per step. The attention mask is what makes this honest: it hides every later position from every earlier one, so the prediction at position 3 cannot peek at position 4 even though both are computed in the same pass.

A six-layer transformer of 4.75 million parameters

The model is nanoGPT's transformer as written, with a configuration sized for this data and this machine.

The whole model. Each block reads the residual stream twice and writes back into it twice; the output head reuses the token table's weights, so scoring a character and reading one use the same 256 numbers.

Model n_layer6 n_head832 dimensions per head n_embd256 block_size256characters of context dropout0.2 vocab_size122 Training batch_size32 max_iters3,000 learning_rate1e-3cosine decay to 1e-4, 100 warmup steps eval_interval250 eval_iters50200 by default, a quarter of an hour of evaluation on this CPU devicecpu compileFalse

Almost all the parameters sit in the blocks. Each block holds four d×dd \times dd×d matrices in attention (queries, keys, values and the output projection) and two d×4dd \times 4dd×4d matrices in the MLP, twelve d2d^2d2 in all, so

N≈12 L d2=12⋅6⋅2562≈4.72MN \approx 12\,L\,d^2 = 12 \cdot 6 \cdot 256^2 \approx 4.72\text{M}N≈12Ld2=12⋅6⋅2562≈4.72M

and the token table adds 122×256=31,232122 \times 256 = 31{,}232122×256=31,232 more, shared with the output head (nanoGPT ties the two, so the same row that reads a character in also scores it on the way out). nanoGPT reports 4.75M and leaves the position table out of its count.

The size came from the next section. Karpathy's Shakespeare model is 10.7M parameters with 384 channels; on this CPU it would have taken seven hours.

Predictions written down before the run

Six numbers, each with where it came from.

The loss at step 0 should be 4.80. A freshly initialised network spreads its probability roughly evenly over the 122 characters, and the cross-entropy of a uniform guess is −ln⁡1122=ln⁡122=4.80-\ln \frac{1}{122} = \ln 122 = 4.80−ln1221​=ln122=4.80. Karpathy does the same check in the video with 65 characters: he expects 4.17, gets 4.87, and reads the gap as a first guess that is less uniform than it should be. nanoGPT starts its weights small enough that the first guess is close to flat, so the gap should be small here, and if it is large, something in the data or the model is broken before training even starts.

The speed comes from a benchmark, and the length of the run from the speed. I timed ten training steps of three model sizes on random integers, which measures the arithmetic and nothing else.

Model Params Batch × context ms per step Tokens/s

README's CPU recipe 0.81M 12 × 64 41 18,519

This run 4.75M 32 × 256 1,287 6,368

Karpathy's Shakespeare model 10.70M 64 × 256 5,052 3,243

The Shakespeare configuration trains on 5,000 steps of 16,384 tokens, 82 million tokens, which at 3,243 a second is seven hours. The mid model's 3,000 steps of 8,192 are 24.6 million tokens, sixty-five minutes at 6,368 a second, plus thirteen evaluations of 100 batches at about half a second each: about 75 minutes in all.

The compute is about 7×10147 \times 10^{14}7×1014 operations. The rule of thumb from the scaling-law papers is that training costs six floating-point operations per parameter per token, two for the forward pass and four for the backward1.

1Kaplan et al., Scaling Laws for Neural Language Models, 2020, section 2.1, which counts the forward pass as about 2N2N2N operations per token and training as three times that.

C≈6 N D=6×4.75×106×24.6×106≈7.0×1014 FLOPsC \approx 6\,N\,D = 6 \times 4.75\times10^{6} \times 24.6\times10^{6} \approx 7.0\times10^{14}\ \text{FLOPs}C≈6ND=6×4.75×106×24.6×106≈7.0×1014 FLOPs

The benchmark says the CPU sustains 6×4.75×106×6,368≈1.8×10116 \times 4.75\times10^{6} \times 6{,}368 \approx 1.8\times10^{11}6×4.75×106×6,368≈1.8×1011 operations a second, about 0.18 TFLOPS. The same run on a rented RTX 4090 would be a matter of minutes.

The data is the limit, and the model is bigger than a compute-optimal one on purpose. The Chinchilla rule says a compute-optimal model sees about twenty tokens per parameter1, which for 3.24 million training characters would be a model of about 160,000 parameters. That rule answers a question about a fixed budget of compute with as much data as you want. Here the data is fixed and compute is cheap, so the usual move is a bigger model trained for several passes over the same text, held back by dropout, keeping the checkpoint with the best validation loss. How many passes are safe has an answer too: repeating data for up to about four epochs changes the loss negligibly against fresh data, and past that each extra pass is worth less until it is worth nothing2. This run makes 7.6 passes, past the safe zone, which is why validation loss and not training loss decides which checkpoint survives.

1Hoffmann et al., Training Compute-Optimal Large Language Models, 2022: model size and training tokens should grow together, and Chinchilla itself is 70B parameters on 1.4T tokens. 2Muennighoff et al., Scaling Data-Constrained Language Models, 2023, with models up to 9B parameters and 900B tokens.

The final validation loss should land between 1.35 and 1.55. The anchor is Karpathy's own number: his 10.7M Shakespeare model reaches 1.4697 in about three minutes on an A100, as the nanoGPT README says, and 1.48 in the lecture. This run has 3.2 times the text, half the parameters, a third of the training tokens and a different language, and those push in opposite directions, hence a band rather than a number.

The cost is a few cents of electricity. The laptop at full load for an hour and a half costs less than the coffee next to it. A rented RTX 4090 costs $0.34 an hour on RunPod's community cloud, which I had checked on 21 October for the Stable Diffusion rental, and it would finish the training in minutes; the half hour of setting up a pod, copying the data and installing would dominate the bill, which would still come to well under a dollar. For a model this size the GPU buys time, and I had the time.

Dropout and the fused attention kernel

Before the long run I ran a five-minute sanity check with the README's CPU recipe: the loss at step 0 was 4.81 against the predicted 4.80, and after 200 steps it was down to 2.3, about what a model that only knows which letter tends to follow which can do. The pipeline worked. Then I launched the real run, and the log said 3.9 seconds per step, three times the benchmark's 1.3. Seventy-five minutes had become three and a half hours.

The log had the clue in it. Every 250 steps nanoGPT stops to evaluate, and evaluation, which runs the same model forward over 100 batches, took about 50 seconds, half a second per forward pass. That is the speed the benchmark predicted. So the model was fast when it evaluated and slow when it trained, and the one thing that differs between those two modes is dropout: the benchmark had run with dropout at 0 and the configuration sets it to 0.2. I stopped the run three minutes in and timed the three possibilities.

Dropout on the residual stream and in the MLP costs nothing measurable. Dropout on the attention weights costs more than the rest of the step put together.

nanoGPT computes attention with PyTorch's scaled_dot_product_attention, which picks a kernel at run time. On a CPU the fast one is a fused, flash-style kernel that never builds the full attention matrix, and asking it for dropout directly shows what happens:

PYTHON Copy

with sdpa_kernel([SDPBackend.FLASH_ATTENTION]): F.scaled_dot_product_attention(q, k, v, dropout_p=0.2, is_causal=True) # RuntimeError: No available kernel. Aborting execution.

The fused CPU kernel refuses dropout, so PyTorch quietly falls back to the generic implementation, which builds the whole 256×256256 \times 256256×256 matrix for each of the 8 heads of each of the 32 sequences and applies the dropout mask to it. Timed on its own, one attention call forward and backward takes 53 ms with the fused kernel and 614 ms with the fallback, 11.6 times more, six layers deep. No warning is printed. The README's own CPU recipe sets --dropout=0.0, because its model is too small to need regularising, and so it never meets this.

The fix is one line of model.py: the attention call gets dropout_p=0.0 and the rest of the dropout stays at 0.2, the residual and MLP dropout that the chart shows to be free. The cost is a slightly weaker regulariser, which matters at 7.6 epochs; whether it hurt shows in the gap between training and validation loss at the end. The restarted run went at 1.7 to 2.0 seconds per step, which is the other prediction that missed: the same configuration that took 1.29 seconds in the cold benchmark took 1.66 twenty minutes later on a hot laptop, and a CPU at sustained load is slower than a CPU for ten steps.

What the model writes

The run finished at step 3,000 with a validation loss of 1.1935 and a training loss of 1.1496, and the last checkpoint was also the best one: validation loss never stopped falling.

The loss on the 197 held-out posts against the predictions written down before the run. The shaded band was the guess for where it would end; it ended below it.

Going back over the predictions one by one:

Prediction Predicted Measured

Loss at step 0 4.80 4.88

Final validation loss 1.35 to 1.55 1.19

Length of the run ~75 min 2 h 02 min

Step time 1.3 s 2.0 s (median)

Cost cents cents

Two of them missed, and in opposite directions. The loss beat its band by 0.16, and 1.19 nats per character is 1.19/ln⁡2=1.721.19 / \ln 2 = 1.721.19/ln2=1.72 bits, the number a SentencePiece run would have to beat. The band was anchored on Shakespeare, and that is where it went wrong: the loss per character of Early Modern English verse and of Spanish prose written for a phone are two different quantities, and I compared them as if model size and data were the only things that changed between them. The gap between training and validation loss says the data limit was looser than I feared, too: 0.01 at step 1,000 and 0.044 at the end, after 7.6 passes over the text, with validation still going down. Somewhere past step 3,000 the model would have started to memorise the posts; this run stopped before it got there.

The time missed by 60%, which is the more instructive failure. The benchmark ran ten steps on a cold laptop with nothing else open; the run lasted two hours on a hot one, while I built the website, rendered figures and trained the SentencePiece tokenizers on the same eight cores. The median step came out at 2.04 seconds against the benchmark's 1.29, and every evaluation took close to a minute. A benchmark is the speed of the machine at its best, and a prediction built on it should carry a margin for the machine being used.

As for what it writes, here are two of six samples, each 700 characters started from an empty line and sampled at a temperature of 0.8, cut to their first few lines:

Sample 1Porque antes de terrando que alguien hace verse algo que es descubriendo que no le he pisto egoístamente de lo que ya eres. Porque no puedo ser medio a que la mirada es que hay que confunderlo por las mujeres que como si hay mujeres con esas feministas que no nos sean quiz. Porque para que nos enseñan una pelea o una gente de su cuerpo machista y suponen en la realidad que habla que tiene nunca a su gota y mal.

Sample 5Aceptar que eres una fracasa. Ese es hermana una posibilidad. Este es un pueblo en seguir diferente. Como una humanidad conseguir. Porque a conocer en un suelo.

Of the 825 words in the six samples, 799 (96.8%) occur somewhere in the training text, and the 26 that do not (terrando, confunderlo, crisidad, masculinación) are built the way Spanish builds words. The shape is the poet's: short paragraphs, anaphora (three lines opening on Porque in the first sample, and further down the fifth a run of lines opening on No), the second person, and the same few subjects coming back, bodies, women and men, love. What it does not have is an argument; a sentence holds together for a few words and then wanders off. That is what 4.75 million parameters looking at 256 characters at a time can do, and it is roughly what Karpathy's Shakespeare model does with Shakespeare.

Before quoting any of it I checked that the model was not reciting. For each sample I searched the training text for the longest run of characters it copies verbatim, and the longest across all six is 26 characters, os hombres con las mujeres, a phrase any Spanish text about gender contains. The model learned how the posts sound, and none of the posts themselves.

1.19best validation loss, 1.72 bits per character, against a predicted 1.35 to 1.55step 3,000 of 3,000, 50 batches of 32 × 256 characters 4.88loss at step 0, against ln 122 = 4.80 for a uniform guessfirst evaluation of the run 2 h 02 minwall time of the run, against 75 minutes predicted from a ten-step benchmarktraining log, steps of 2.04 s median 2.3×step time with dropout inside attention on this CPU3,751 ms against 1,656 ms, mean of 5 steps 26longest run of characters in six samples copied verbatim from the training textsamples of 700 characters at temperature 0.8

The code, without the corpus or the weights, is on GitHub as poet-gpt: nanoGPT with the one changed line, the scripts that measured everything above, and the figures. Pointed at any folder of text files it repeats the whole run, predictions included. The next run is the one this left open, the same model on a 4,000-token SentencePiece vocabulary, compared at last on bits per character.