aboutsummaryrefslogtreecommitdiff
path: root/bench/benchstat.py
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-06-27 00:58:05 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-06-27 01:03:41 +0200
commitaf55803d0dff9eb692c760c8d486e736c945061d (patch)
tree0875a1112bf98304e445e0544c4309670d6f4769 /bench/benchstat.py
parent0f861b7f3098750ddee4f6570f3be228d1da5320 (diff)
downloadffwd-af55803d0dff9eb692c760c8d486e736c945061d.tar.gz
ffwd-af55803d0dff9eb692c760c8d486e736c945061d.zip
chore: clean up comment formatting and wording
Diffstat (limited to 'bench/benchstat.py')
-rwxr-xr-xbench/benchstat.py21
1 files changed, 14 insertions, 7 deletions
diff --git a/bench/benchstat.py b/bench/benchstat.py
index 7f8b695..8036c97 100755
--- a/bench/benchstat.py
+++ b/bench/benchstat.py
@@ -4,13 +4,20 @@
usage: benchstat.py OLD.json NEW.json [--alpha 0.05] [--threshold 3.0]
-Per benchmark: median ns/op of both sample sets, the delta, and a
-Mann-Whitney U significance test. A delta only counts when it is BOTH
-significant at --alpha AND at least --threshold percent: back-to-back
-runs of an unchanged binary drift a couple of percent with machine state
-(thermals, core scheduling), so smaller shifts are below the
-reproducibility floor even when statistically clean. Everything else
-prints as "~" - treat it as noise, not improvement or regression.
+Per benchmark:
+median ns/op for both sample sets, the delta, and a Mann-Whitney U
+significance test.
+
+A delta counts only when it is both:
+- significant at --alpha
+- at least --threshold percent
+
+Back-to-back runs of an unchanged binary can drift a few percent with machine
+state, such as thermals and core scheduling. Smaller shifts are below the
+reproducibility floor even when statistically clean.
+
+Everything else prints as "~"; treat it as noise, not improvement or regression.
+
Exits 1 if any benchmark regresses past both bars.
"""