ML//inference

Inference is the stage of a machine learning model's life in which the trained model runs on current inputs to produce predictions or generations, and it is what every request to a deployed model costs. The weights are used rather than learned, although the surrounding application may retrieve information, update memory or call tools between invocations (RAG retrieves documents before the call so the model conditions on them as ordinary context).

Inference is the stage of a machine learning model's life in which the trained model runs on current inputs to produce predictions or generations, and it is what every request to a deployed model costs. The weights are used rather than learned, although the surrounding application may retrieve information, update memory or call tools between invocations (RAG retrieves documents before the call so the model conditions on them as ordinary context).

For an autoregressive LLM, the transformer produces logits for the next token, sampling selects a continuation, and that token becomes input to the next step; the KV cache keeps the attention keys and values of earlier tokens so they are not recomputed at every step. The economics differ sharply from training: one training run creates weights, while inference may reuse them across millions of requests, so repetition, latency and utilization make its efficiency a product constraint (inference economics). A system may also choose to spend much more than one pass per token on a hard request, thinking longer, sampling several answers or searching with a checker (test-time compute); the answer improves and the weights stay exactly as they were.

Training builds the instrument. Inference is every performance, including the electricity bill and the waiting audience.