// System Design · 04
Latency vs Throughput
Latency: one requestleads toThroughput: all requests
How fast one thing is, versus how much fits through per second.
How it fits together
- Clients uses / calls Request queue (1 request in)
- Request queue uses / calls Worker pool (2 dequeue)
- Worker pool uses / calls Clients (3 response)
Spot it
- Average looks fine but users still complain
- Batching helped the job but hurt the API
- Load tests pass, production feels slow
It costs
Batching and queues raise throughput but add latency to each item.
Go deeperThe full explanation and what to ask for · 1 min readExpandCollapse
Latency is time per request; throughput is requests per unit of time. A highway can be wide (throughput) and still slow (latency).
Averages hide pain. Track percentiles: p99 is what your unluckiest 1% feel, and at scale that is a lot of people.
Little's Law ties them together: items in the system = throughput × latency. Push throughput near capacity and queues, and therefore latency, explode.
Ask for it
- p50 / p95 / p99
- tail latency
- requests per second
- batching
- backpressure
Measure this endpoint's p50, p95 and p99 latency and its max sustainable requests per second, then tell me which one is limiting us and why.