infrastructure//throughput

Throughput is the amount of useful work completed per unit of time: requests per second, tokens per second or jobs per hour. It describes aggregate capacity, not how long one request waits.

Throughput is the amount of useful work completed per unit of time: requests per second, tokens per second or jobs per hour. It describes aggregate capacity, not how long one request waits.

Any claim of twice the performance is incomplete until the numerator, workload and latency constraint are named. Batching amortizes overhead and raises accelerator utilization, so higher batch sizes may increase token throughput while violating an interactive latency target. A benchmark can also report model tokens per second or total served tokens per second, which answer different questions.

Advice26 · sep 04Read noteAlways ask: more of what, under which waiting-time promise? A larger batch can double tokens per second while breaking the latency an interactive user expects, so a figure without its numerator and its constraint compares nothing.