infrastructure//latency

Latency is the elapsed time experienced by one unit of work. For generated responses, useful measures include time to first token and the delay between subsequent tokens.

Latency is the elapsed time experienced by one unit of work. For generated responses, useful measures include time to first token and the delay between subsequent tokens.

Latency is distinct from throughput, and systems often trade one for the other: a server may process more total tokens by batching requests while making each user wait longer. Tail latency also matters: a pleasant average can hide a small fraction of painfully slow requests that dominate user experience.

Throughput fills the restaurant. Latency is how long your own plate takes to arrive.