The benchmark lied because my laptop was plugged in

Local benchmarks look more serious than they are.

You run one command:

wrk -t12 -c400 -d30s http://localhost:3000/ping

It prints a clean table, and suddenly the result feels official.

Then you unplug the laptop, run it again, and the numbers move.

Not always by a lot. But enough to remind you that the benchmark did not happen in some perfect lab. It happened on your desk, with Chrome open, battery settings doing their thing, maybe music playing, maybe Spotlight also deciding this is the right time to work.

My very professional benchmark setup is just a MacBook Pro and too many tabs.

plugged in matters#

Laptops change behaviour depending on power and heat. The CPU may boost differently on battery. The fans may already be working. The machine may be warm from a previous run.

So if I compare two frameworks and the difference is tiny, I should be careful.

If one run says:

framework A: 90,400 req/sec
framework B: 89,900 req/sec

I would call them basically tied.

Small differences are easy to fake by accident. Big repeated differences are easier to trust.

one run tells too little#

The first run can be strange. The machine is cold, the process is new, caches are in a different state, and the runtime may not have settled.

Then the second run improves.

Then after a few runs the laptop gets warm and the numbers drop a little.

So I like seeing repeated runs more than one clean number:

run 1: 91k
run 2: 93k
run 3: 92k
run 4: 91k
run 5: 92k

That gives me more confidence than:

result: 93,482.11 req/sec

The second one looks precise, but precision can be cosmetic.

the command is part of the result#

wrk -t12 -c400 -d30s is a setting with opinions.

Change the concurrency and the result can change. Run for 5 seconds instead of 30 and startup effects matter more. Run the client and server on the same machine and they compete for CPU.

That is why I like posting the exact command in benchmark posts. The benchmark stays imperfect, but the reader can see what kind of pressure was applied.

For Go microbenchmarks, I also like this:

go test -bench=. -benchmem -count=5

That -count=5 is a small thing, but it saves you from building an opinion around one lucky run.

variance is useful#

If one run is fast and another run is weirdly slow, I pause before calling it noise. It may be the thing worth investigating.

Maybe GC kicked in. Maybe allocations changed. Maybe a lock got hot. Maybe the benchmark tool was also overloaded.

This matters more for latency than throughput. A nice average can hide requests that were painful. Users experience one request at a time.

I still like local benchmarks. They are cheap and they answer “is this obviously bad?” very quickly.

I trust them more when they come with context.

If I publish a benchmark, I should say the machine, the command, the number of runs, and what the benchmark leaves out. Also whether the laptop was plugged in, because apparently that is part of science now.