ML//inference//local inference

2026-10-23

Local inference is running a language model on one's own machine (a laptop, a workstation, a server in the building) instead of calling a provider's API, and it is chosen when data must not leave the machine, when each request should cost nothing, or when the system must work offline. It needs an open-weight model, a file that holds it (GGUF in the llama.cpp family), an engine that runs it (llama.cpp, Ollama, LM Studio, vLLM) and enough memory for the weights and the KV cache.

Two numbers of the machine decide the experience. Capacity decides which model fits: the weights at their bits per weight, the cache for the reserved context and room for the operating system. Memory bandwidth decides how fast it writes, because every generated token reads every active weight once (memory-bound). A laptop with 64 GB of unified memory at 128 GB/s holds a 35-billion-parameter model and writes about 12 tokens a second with a mixture of experts that reads 2 GB per token, and 2.6 with a dense 27-billion one that reads 16 GB.

Before downloading a model, divide. The weights in GB are the parameters in billions times the bits per weight over eight, and the writing ceiling is the bandwidth over the GB read per token. A model that fits but writes below reading speed (5 to 8 tokens a second) is one that ends up unused.

Quantization to about 4.9 bits per weight (Q4_K_M) is the usual default: it brings the file under a third of its 16-bit size and roughly triples the writing speed.

Reading long prompts (prefill) is limited by arithmetic, which is where an integrated GPU is weakest and a discrete one with its own VRAM strongest.

Work that needs a lot of arithmetic for a long time (evaluations over thousands of cases, batch processing, fine-tuning) usually goes to a rented GPU, and the local machine keeps the private and interactive work.