ML//inference

Inference runs a trained model with current inputs to produce predictions or generations. The weights are used rather than learned, although the surrounding application may retrieve information, update memory or call tools between invocations.

Inference runs a trained model with current inputs to produce predictions or generations. The weights are used rather than learned, although the surrounding application may retrieve information, update memory or call tools between invocations.

For an autoregressive LLM, the transformer produces logits for the next token, sampling selects a continuation, and that token becomes input to the next step. The economics differ sharply from training: one training run creates weights, while inference may reuse them across millions of requests.

Training builds the instrument. Inference is every performance, including the electricity bill and the waiting audience.