TokenWalk: a step-by-step simulator of local LLM inference

The console behind the transformers article. Set up a run, press Play and watch the request go from the disk to the last token; click anything to see what it is. It also tells you which models fit on which machines, and can run the same request on your own Ollama.

While writing Transformers and local inference I kept coming back to one question. On my laptop, a model with 35 billion parameters takes about a minute before it writes the first word, and then it writes twelve words a second. Why a minute? And why twelve?

The article answers it one idea at a time, each with a small figure and a single slider. That is how I learned it, but it hides the interesting part: in a real request everything happens at once. The prompt you wrote decides how big the cache gets, the cache fights the model for the same memory, and the speed of that memory decides how fast the answer comes out.

So I built TokenWalk. You pick a machine, a model and a prompt, press Play, and watch the request travel from the disk to the last word of the answer. You can pause at any moment and click on anything to ask what it is. Open it here; it runs in the browser and there is nothing to install. The code is on GitHub.

The default request

By default it plays the article's example. An assistant for doctors sends the model a short system prompt, the history of a made-up patient (about 5,650 tokens), a question of 50 tokens, and room for an answer of 300. The model is Qwen3.6 35B-A3B, a mixture of experts, compressed to Q4_K_M, a bit under five bits per weight. The machine is my ThinkPad, whose only graphics chip is the Radeon 860M built into the processor, sharing 64 GB of unified memory with everything else.

Three things happen, and the console times each one:

Load. If the model is not in memory yet, 21.7 GB have to come off the SSD.

Prefill. The model reads the prompt, a step called prefill, in blocks of 512 tokens. Each block climbs all 40 layers together, so one read of the weights serves 512 tokens at once, and the limit is how fast the GPU can do arithmetic. Thirteen blocks take about 76 seconds.

Decode. Now the answer comes out one token at a time. Every token means reading the active weights again, 1.84 GB, and doing very little arithmetic with them. The GPU spends most of its time waiting for memory, so the speed is set by memory bandwidth: about twelve tokens a second.

That is local inference in two numbers. Reading the prompt is limited by compute; writing the answer is limited by memory. Most of the console exists so you can see that happen instead of taking my word for it.

The screen and its five views

During prefill. While a run plays, the setup is locked; only the sampling settings stay live.

The screen has four parts. On the left you set up the run: machine, model, prompt size, and the settings of the engine. On the right, Monitor keeps the numbers you would care about as a user: how long until the first word, how fast it writes, how much memory it takes, and what is slowing it down right now. Along the bottom there is Play, Pause and a timeline you can drag, from the whole request down to a single operation inside one layer.

The middle shows the same moment in five views.

The System view draws the computer: the memory chips, the SSD, the processor with its CPU, NPU and the eight compute units of the integrated GPU, and the bytes moving over the memory bus. During prefill the compute units are full. During decode they sit mostly idle, waiting for bytes. If you only look at one thing, look at that.

The Model view draws the model as a grid: tokens across, layers up. Each token climbs its column through every layer. When a cell in an attention layer finishes, it turns yellow, because it left its keys and values behind for the tokens that come later. That yellow area is the KV cache. Cells in the Gated DeltaNet layers turn grey instead: those layers keep no memory per token and fold everything into one small state, drawn at the end of the row.

The grid during decode. The KV cache is the yellow part.

Switch the grid to Matrices and you see one layer to scale: its matrices, the order in which they are used, and the 256 experts of the layer with the eight that the router picked for this token lit up.

One layer: what it reads, in what order, and which experts this token woke up.

The Memory view starts with the whole 64 GB, split into the model, the cache, the other programs and what is free. Then it zooms into the cache, a row per layer and a column per group of tokens, and a magnifier follows your mouse down to single tokens. Watch the cache grow one column per token as the answer is written.

From 64 GB down to one token.

The Tokens view shows the prompt as it is read and the answer as it is written. Click any word of the answer and you see how it was chosen: the raw probabilities from the logits, then temperature, then top-k, then top-p, with the candidates each step keeps and throws away.

How one word was chosen, step by step.

The fifth view, Flow, draws the shapes of the data instead of the machine, and it has a section of its own further down.

Explainers and close-ups

Everything on screen can be clicked. You get a short explanation of what it is, why it matters, what would change it, and its value right now. From there you can jump to the same thing in another view, to the setting that controls it, or to the wiki note that explains the idea properly.

What you get when you click the KV cache.

Some things also open up. A matrix opens to scale, a row of it becomes the single dot product it costs, and a single weight shows its bits. At Q4_K_M one weight is four bits plus its share of a scale stored once for every 32 weights: 0.61 bytes. From there you can multiply your way back up. Times 2,048 is a row of 1.3 KB; times 8,192 rows is a matrix of 10 MB; a layer with all its matrices is about 512 MB; forty layers and the embeddings make the 21.7 GB that had to come off the disk. Every size in the console is built like that, so you never have to trust a number you cannot trace.

One weight, and the multiplications that turn it into a model.

The deep timeline

The timeline along the bottom has an Explore button, and pressing it grows the bar into the whole screen. It is the same request on one axis of time, and every bar opens into the bars that make it up as you zoom, from the whole request down to the rows of a single matrix. On the laptop one row of a matrix lasts a few tens of nanoseconds, and the scale at the top tells you how long one chunk, one pass, one layer and one operation take in the run you are looking at.

Each bar is as long as the cost model says it should be: the bytes it reads divided by the bandwidth the machine reaches, or its arithmetic divided by the compute, whichever is larger, plus a fixed cost per layer for launching the work. Under the bars, two strips show how much of the memory bandwidth and of the compute each slice uses. Zoom from a prefill chunk into a decode pass and you watch the compute strip go from full to nearly empty while the memory strip stays high, which is the whole argument of the transformers article in one movement of the mouse wheel.

Click a bar and the side panel says what it does and why it takes that long, with its bytes and its arithmetic. A bar inside a layer also has a button that opens the same operation in Flow.

One round of decode, zoomed to the experts of one layer. The panel explains why it takes 1.07 ms.

Tensor shapes in the Flow view

The other views tell you where the bytes are and how long they take. Flow answers a question they skip: what shape the data has at each step of a pass, and why those shapes fit together. It draws one pass as a strip of rectangles, left to right, wrapping onto a second row like a line of text. The height of every tensor is the number of tokens the pass carries, so a prefill chunk of 512 tokens is a tall block and a decode step is a single thin row. Its width is the numbers each token has at that point. The sizes on screen go from 1 to 248,320, so lengths are on a log scale, and the view says so at the top: each doubling adds one notch.

Each dimension keeps one colour everywhere. The hidden width of Qwen3.6, 2,048 numbers, is blue on every edge where it appears; the 16 query heads are salmon and the 2 KV heads are pink. A matrix product is drawn the way it is taught on a blackboard, with the weight above its result and the input to its left. The dimension the two share then shows up twice, at the same length and in the same colour, and is gone from the result. The card at the bottom writes the product out with that shared dimension crossed out, so the query projection reads as a row of 2,048 times a matrix of 2,048 by 8,192, giving a row of 8,192. Weights are violet, lighter the more bits each weight keeps. They are never coloured by their values, which the console does not have.

The scores of one attention head during decode. The keys are the cache itself, as wide as the context so far.

Attention is where the drawing earns its place. The cached keys are the second operand of the scores: a slab as wide as the context, which grows by one column with every token, inside a dashed outline of the room that num_ctx reserved. The cached values stand on their end above the weighted sum. A small diagram above the query shows the grouping that makes the cache small, sixteen query heads sharing two sets of keys. In a prefill chunk the scores carry a dark corner, the causal mask that stops each token from seeing the ones after it.

Qwen3.6 alternates two kinds of layer, so the layer is drawn with two lanes: DeltaNet above, attention below, and the lane the current layer does not use is dimmed. The DeltaNet lane has no cache at all. Its memory is a 128 by 128 state per head, drawn as a square that keeps its size whatever the length of the prompt. The feed-forward half of the layer shows the 256 experts as a grid with the ones this token woke lit, and one of them drawn to scale beside it.

One decode pass of Qwen3.6, from the token ids to the sampled token.

Under each row, a thin line per tensor runs from the operation that makes it to the last one that reads it. Almost all of them live for a few operations and are freed. The KV cache and the DeltaNet state end in an arrow instead, because they outlive the pass. The frame of the layer is striped in the colour of the memory it lives in, which is the detail that matters on a card too small for the whole model: there the first layers show up in system RAM and the rest in VRAM.

The view follows the playhead, so while the request plays a highlight moves through the strip and the weight being read fills up row by row. With the arrow keys you walk one pass step by step instead, from the embedding to the sampled token, with one sentence per step and the camera easing to that part. Clicking any tensor opens the usual explainer with its shape, its size, the operations that make and read it, and how long it lives. Two things in it are illustrative. Which experts light up is a fixed pseudo-random choice, because the console does not run the router, though how many of them are read is real. And Q4_K_M mixes two formats per tensor in a way the model's configuration does not record, so every weight is drawn at the 4.9-bit average.

Preset scenarios

Each of these links opens the console already set up:

A cold start. The model has dropped out of memory, so before reading the prompt it has to load 21.7 GB again. This is what happens if your requests are further apart than Ollama's keep_alive.

Ollama's default context on this laptop. It reserves room for 262,144 tokens, 5.4 GB of memory, for a conversation of 7,000. Look at the Memory view.

A dense 27B with a small draft model, against the mixture of experts with the same draft. The same trick nearly doubles the speed of one and slows the other down.

Pick GPT-3 at full precision on the laptop. It needs 360 GB, and the console tells you before you press Play, with versions that do fit.

A setup that cannot run says so before you start.

Estimates for 67 machines

This is where the console became useful beyond the article. The catalogue has 67 machines, from Apple laptops to gaming GPUs to datacenter cards, and 22 models, each with the source of its numbers. The What fits tab runs your request on every combination: whether it fits in memory, how long until the first word, how many tokens a second, and what limits it. Click a row and that machine and model are loaded.

Only my laptop is measured. Every other machine is estimated from its published specs, with a correction for each hardware family learned from public llama.cpp benchmarks. I checked the estimates against 185 of those benchmarks, leaving each machine out of its own correction so the test is fair. For the writing speed, half of them land within 7%; for the time to the first word, within 15%. Intel GPUs and datacenter cards are the weak spots, because there are few public numbers to learn from. The whole comparison is in the accuracy report.

In the plot each point is one of those benchmarks, with the published speed across and my estimate up; a point on the diagonal is exact, and the dashed lines are 20% either side. Decode stays close to the line from five tokens a second to three hundred, apart from a few NVIDIA and Intel points. Prefill spreads more, and most of the points far off the line are AMD and Intel.

Each point is one published llama.cpp benchmark, estimated with its machine left out of its family's correction. The data is data/benchmarks.json and data/calibration.json in the TokenWalk repository, drawn by tools/accuracy-plot.mjs.

Live runs on Ollama and llama.cpp

A simulation explains things, but it does not prove them. So the console can also send the request to a model running on your machine, through Ollama or llama.cpp's server, and show what it measured next to what it predicted. It reads the real shape of your model from its GGUF file, times every token as it arrives, and the views follow the real run.

Connected to the laptop's Ollama.

On my laptop, with the model not yet loaded and a short question:

Estimate Measured

Load 54 s 70 s

First token 55 s 71 s

Writing, 32 tokens 13 tok/s 11 tok/s

Total 57 s 74 s

The first real runs taught me something the console did not know. The tokens were not arriving one by one; they came in little bursts of two or three. Qwen3.6 carries an extra layer trained to guess one token ahead, and Ollama uses it by default to draft two tokens per round, a trick called multi-token prediction. At temperature 0 it took the laptop from 13.2 to 15.6 tokens a second. At 0.7 it did the opposite and dropped it to between 10.6 and 11.5, because with looser sampling more of the guesses get thrown away. The console now models it, with acceptance rates fitted to those runs.

After a real run: the estimate, the measurement, and a button to calibrate one to the other.

If you run it on your own machine, you can send me the result: the console fills in a GitHub issue with your machine, model and timings, and once I accept it, it shows up in What fits as a real measurement next to the estimate.

To connect from this site, Ollama has to allow it. A copy opened from localhost works as it is; for infraphysics.net, add it to OLLAMA_ORIGINS and restart Ollama:

Windows: quit Ollama from the tray, run setx OLLAMA_ORIGINS "https://infraphysics.net", then start it again.

macOS: launchctl setenv OLLAMA_ORIGINS "https://infraphysics.net", then restart the app.

Linux: sudo systemctl edit ollama.service, add Environment="OLLAMA_ORIGINS=https://infraphysics.net" under [Service], then sudo systemctl daemon-reload && sudo systemctl restart ollama.

If something is wrong, the console says what and shows the line for your system.

Provenance of the numbers

MeasuredMy ThinkPad, from real runs.

Read from the filesThe shape of every model (layers, heads, experts, which layers keep a cache), so the bytes per token and the cache size are exact.

EstimatedEvery other machine, marked as an estimate wherever it appears.

Made upThe text of the answer, the logit values and the values of individual weights in the close-ups. The counts, sizes and times around them are computed.

Left outSeveral requests at once, other programs competing for the machine, and heat.

Implementation

TokenWalk is TypeScript built with Vite, with no framework and nothing to install. Almost everything is drawn on a canvas, frame by frame, and the build packs it all into a single HTML file that also opens straight from disk. The calculation is one plain function, estimate(), that takes a machine, a model and a request and returns the timings and the memory; the views only draw what it returns.

Most of the work went into the small things a first-time reader gets wrong: a label that cuts off, a colour that means two things, a number with nothing to compare it to. I check those with scripts that drive a headless Chrome through every view at two window sizes and fail if anything breaks.

To run it yourself:

BASH Copy

git clone https://github.com/yago-mendoza/tokenwalk cd tokenwalk npm install npm run dev

npm run build writes the single file to dist/tokenwalk.html. Issues and measurements are welcome on GitHub.