diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-06-27 00:58:05 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-06-27 01:03:41 +0200 |
| commit | af55803d0dff9eb692c760c8d486e736c945061d (patch) | |
| tree | 0875a1112bf98304e445e0544c4309670d6f4769 /bench/benchstat.py | |
| parent | 0f861b7f3098750ddee4f6570f3be228d1da5320 (diff) | |
| download | ffwd-af55803d0dff9eb692c760c8d486e736c945061d.tar.gz ffwd-af55803d0dff9eb692c760c8d486e736c945061d.zip | |
chore: clean up comment formatting and wording
Diffstat (limited to 'bench/benchstat.py')
| -rwxr-xr-x | bench/benchstat.py | 21 |
1 files changed, 14 insertions, 7 deletions
diff --git a/bench/benchstat.py b/bench/benchstat.py index 7f8b695..8036c97 100755 --- a/bench/benchstat.py +++ b/bench/benchstat.py @@ -4,13 +4,20 @@ usage: benchstat.py OLD.json NEW.json [--alpha 0.05] [--threshold 3.0] -Per benchmark: median ns/op of both sample sets, the delta, and a -Mann-Whitney U significance test. A delta only counts when it is BOTH -significant at --alpha AND at least --threshold percent: back-to-back -runs of an unchanged binary drift a couple of percent with machine state -(thermals, core scheduling), so smaller shifts are below the -reproducibility floor even when statistically clean. Everything else -prints as "~" - treat it as noise, not improvement or regression. +Per benchmark: +median ns/op for both sample sets, the delta, and a Mann-Whitney U +significance test. + +A delta counts only when it is both: +- significant at --alpha +- at least --threshold percent + +Back-to-back runs of an unchanged binary can drift a few percent with machine +state, such as thermals and core scheduling. Smaller shifts are below the +reproducibility floor even when statistically clean. + +Everything else prints as "~"; treat it as noise, not improvement or regression. + Exits 1 if any benchmark regresses past both bars. """ |
