You’re (probably) measuring application performance wrong. Humans have a strong bias for throughput. "I can handle X requests per second." Real capacity engineers use response-time curves.
It all comes down to queueing theory. Unfortunately, computers don’t degrade gracefully under load. 70% CPU is smooth sailing. 95% is a nightmare. Programmers (incorrectly) focus on the absolute value, when really they should be looking at the derivative.
Highways are the perfect real life example of this. Traffic engineers study flow density, not overall vehicle counts. A road handling 10,000 cars per hour (throughput) means nothing if the average speed drops to 5mph (response time). Computers are the same.
Every CS student knows Big-O complexity. Which sorting algorithm to use, etc. Yet queuing theory and resource utilization are (at best) a theoretical blip in undergrad OS courses. It’s pretty straightforward, but almost *no one* uses the right units IRL. Next time you
@lauriewired People are pretty good at using percentiles these days but will often fudge which percentile they’re talking about, p99 is a sigma harder than p95. One thing I almost never see is that queues can be modeled loosely as a poison distribution and then approximated with a log normal
@lauriewired This is partially why Performance engineers use warm up periods and tests from cold start and tests under load.... same reason network buffer bloat tests work the way they do
@lauriewired Cool, it is like over exposure in a photo that clips the highlight 🤔
@lauriewired This reminds me of the four golden signals from Google’s SRE book: 1. Latency 2. Traffic 3. Errors 4. Saturation Looking at a subset of these metrics isn’t sufficient for monitoring. But if all four were healthy, you could be confident that things were good.
@lauriewired I suggested this answer already on LinkedIn, but here, there's a much different audience so.. Don't forget to mention Coordinated Omission: measuring response time is not trivial at all too! See https://youtu.be/lJ8ydIuPFeU?s... for some "oh sh*t" moment!
@lauriewired So response time should be metric
@lauriewired Throughput brags hide pain in the tail. Who cares about X rps if p99 stalls. Users feel the slowest percentiles, not the average. Track p95 and p99, queue depth, and variance, then use backpressure, timeouts, and load shedding to keep the curve flat.
@lauriewired It’s very funny when I talk with one of my engineers and they tell me “we’re handling 50k TPS.” - yes, sure, now show me the 99th percentile response time and we can truly evaluate things.
@lauriewired one of my old profs in college has great resources on using queuing theory for system performance analysis https://www.cs.cmu.edu/~harcho...
@lauriewired Throughput's a trap response time curves tell the real story. What's your apps actual performance bottleneck by the way ?
@lauriewired This why gaming benchmarks have shifted from measuring pure FPS to frame times and consistency.
@lauriewired Lol :))) what if you don't need a response? REAL engineers use 'sustained throughout' ie max constant throughout you can handle w/o queueing anything... Plus inter arrival distribution to analyze real flow
@lauriewired Any resource to read about it?
@lauriewired Ask performance testing engineers not the code monkeys who do the CRUD and Kafka slop
@lauriewired Related classic paper: https://dl.acm.org/doi/10.1145...
@lauriewired @threadreaderapp unroll
@lauriewired Same problem in Internet bandwidth. People buy throughput like 1 G. But low latency makes it amazing and useful.
@lauriewired @antonvputra does comparatives between languages/libraries/frameworks. He measures how Response times and CPU/RAM usage behaves when the services are put to their breaking point. Very interesting videos.
@lauriewired Is there a good way to talk about it? The metrics vary wildly depending on the inter-arrival distribution. How do you pitch your results without multi-page appendix on how the traffic was modelled?
@lauriewired Really helpful in an engineering field where latency correlates with the braking distance of a car.
@lauriewired This honestly reminds me of Factorio and Satisfactory. Very good insight so thank you.
@lauriewired time (command)
@lauriewired Latency is king.
@lauriewired Many things happen between request > response and each can be measured and tuned individually.