ML//inference//inference batching

Inference batching is the grouping of several inference requests into one pass, so that the accelerator can execute larger parallel operations and amortize scheduling and memory overhead.

Inference batching is the grouping of several inference requests into one pass, so that the accelerator can execute larger parallel operations and amortize scheduling and memory overhead.

Static batches wait for a fixed group. Continuous batching inserts and removes sequences as generation proceeds, which better matches requests that finish at different times. Larger batches usually improve utilization until memory or queueing becomes the limiting factor: each active sequence adds its own activation and KV-cache memory to the model memory footprint, and waiting for a batch to form can worsen each request's latency even as the server grows more efficient.

The machine likes a crowded bus. The passenger cares when it departs.