One number is not a measurement¶
Level: 301 · deep dive
Stub — an outline, not a lesson. There is no runnable example behind this page yet, so nothing on it has been through the check that backs every other claim in this library. The bullets below are the questions the finished page has to answer.
One line: A benchmark's result is a distribution, and its minimum, its median and its tail answer three different questions — choose the question before you choose the number.
The questions this page has to answer¶
- Why the minimum is the usual answer for a microbenchmark — the talk: "Noise only makes things slower" — and when it is the wrong one. The talk's distribution has a minimum of 1.77 ns per element, a median of 2.10, a mean of 2.31 and a 99th percentile of 3.50.
- Best case, typical, or bounded worst case: the talk says pick one. Which kind of system needs which?
- What you can do about noise — pin the process to a core, isolate the core, fix the frequency, move interrupts elsewhere — and what is left after all of it.
- Sensitivity: is this code dominated by the branch predictor, the cache or the scheduler, and how do
perfand the shape of the distribution tell you? - Why a benchmark can say faster while the application says no difference, and what timing inside the real code path gives you that a microbenchmark cannot.
- Benchmarking — how fast? — against profiling — why that fast? — and the tools for each: Google Benchmark and Catch2,
perf, Tracy, Callgrind, uops.info.
In the talk¶
So, what can we do? and Conclusion (slide source ↗).