ML//inference//inference economics

Inference economics is the accounting of what it costs to turn a trained model into repeated useful outputs, and it decides whether to pay an API per use or to own or reserve serving capacity. API pay-per-use converts capacity into a variable expense (OpEx, alongside energy and operations) and is excellent when demand is uncertain or operational simplicity matters.

Inference economics is the accounting of what it costs to turn a trained model into repeated useful outputs, and it decides whether to pay an API per use or to own or reserve serving capacity. API pay-per-use converts capacity into a variable expense (OpEx, alongside energy and operations) and is excellent when demand is uncertain or operational simplicity matters.

At sustained scale, owning or reserving serving capacity may reduce unit cost, but only if utilization justifies the fixed commitment: owned accelerators turn expected future usage into an upfront CapEx, and only high throughput spreads that fixed cost across enough completed work. Model quality, latency targets, batching, memory footprint, engineering labor and idle hardware all belong in the comparison. Price per token is a useful line item, not the whole business case.

For a product the estimate starts before any hardware question. A scope document says which features call the model; each feature gets a token estimate (input plus output per call, calls per user per day); and the smallest model that meets the quality bar on a test set is chosen, because the price per token of a frontier model and a small one can differ by fifty times. Serving is a different line from the bill of producing the model (training cost), and for a product that succeeds it is the larger one.

Renting removes the empty room. Owning becomes interesting only when you can keep the room busy.