ML//inference//local inference//llama.cpp

llama.cpp is an open-source inference engine, written in C and C++, that runs language models stored as GGUF files on CPUs and GPUs, and it is the code most local inference runs on: Ollama and LM Studio are built on it or on its tensor library, ggml. It is used directly when every option of the engine is needed.

llama.cpp is an open-source inference engine, written in C and C++, that runs language models stored as GGUF files on CPUs and GPUs, and it is the code most local inference runs on: Ollama and LM Studio are built on it or on its tensor library, ggml. It is used directly when every option of the engine is needed.

It is compiled for its target: CUDA for NVIDIA, Metal for Apple, Vulkan for most other GPUs (an AMD integrated GPU, for instance) or plain CPU. Its server, llama-server, speaks an OpenAI-compatible HTTP API and exposes what the wrappers hide: a draft model for speculative decoding (--model-draft), saving and restoring a slot's KV cache on disk (--slot-save-path), the chunk size of prefill, the cache type and the number of parallel slots.

Ollama is to llama.cpp what an ORM is to raw SQL: convenience and managed models on one side, every option and no management on the other. A project that needs a draft model or caches saved to disk moves to the llama.cpp server.

A slot is one conversation's state on the server, with its own KV cache. Saving a slot and restoring it later turns a minute of prefill over a long fixed document into a fraction of a second read from the SSD; for hybrid models the slot must also hold the fixed state of the linear-attention layers.

Prefill runs in chunks of 512 tokens by default, which is why its time rises in small steps rather than smoothly.