ML//inference//local inference//Ollama

Ollama is a local model server built on llama.cpp's tensor library: a background process that downloads open-weight models, loads them into memory and answers HTTP requests on `localhost:11434`, so that terminals, scripts, chat interfaces and agents can use a local model as they would use an API. It is the easiest entry to local inference, and it hides most of the engine's options in exchange.

Ollama is a local model server built on llama.cpp's tensor library: a background process that downloads open-weight models, loads them into memory and answers HTTP requests on localhost:11434, so that terminals, scripts, chat interfaces and agents can use a local model as they would use an API. It is the easiest entry to local inference, and it hides most of the engine's options in exchange.

It works like a database server with its clients. ollama run, the ollama Python package, Open WebUI or an MCP bridge for Claude Code are all clients, and installing the Python package installs no model. Models are managed like container images (pull, list, rm) and stored as GGUF blobs. A model goes through pull (download, once), load (disk to memory, 55 seconds cold for a 21.7 GB model on a laptop SSD), serve, idle and unload, which happens when its keep-alive expires.

Settings live in two places. Environment variables are read when the server starts, apply to every request and need a restart of the server; options sent in a request (num_ctx, keep_alive, temperature) apply to that call only. Setting a variable in a terminal does nothing to a server that is already running.

The variables that matter most on a laptop are OLLAMA_CONTEXT_LENGTH (the default context; without it Ollama sized the context from the GPU memory it detected and reserved 262,144 tokens on a 64 GB machine), OLLAMA_FLASH_ATTENTION, OLLAMA_KV_CACHE_TYPE (KV cache quantization), OLLAMA_KEEP_ALIVE and, on AMD integrated GPUs, OLLAMA_IGPU_ENABLE, without which it ignores the GPU and runs on the CPU.

ollama run <model> --verbose prints the load time, the prompt eval rate (prefill) and the eval rate (decoding); ollama ps shows the loaded size in GiB, the CPU and GPU split and the reserved context. The API returns the same durations in nanoseconds.

It exposes neither speculative decoding with a draft model nor saving the KV cache to disk; for those the llama.cpp server is used directly.