Repository navigation
fix(bench): require a 5% margin before reporting a cold-build regression - #2002
Merged
Merged
Conversation
The nightly gate compared each board's fbuild/PlatformIO cold ratio against the trailing 7-day median and warned on any excess. A median sits in the middle of its distribution, so the rule fired by construction roughly half the time: replaying benchmark-stats:history.jsonl gives 53% (uno) and 35% (esp32s3) over the full history, and 50%/33% across the quiet 2026-11-08..12 nights. The 2026-11-12 run (5360eae) hit exactly that: uno 0.898 vs a 0.896 median, a 0.2% miss, on a night when fbuild was 3.3x faster than PlatformIO on ESP32-S3 and ~10% faster on the Uno. A gate that cries wolf on noise trains everyone to ignore the annotation. Require `current > median * (1 + REGRESSION_MARGIN)` with a 5% margin: - measured night-to-night spread in the current regime is +-1.5% (uno 0.879-0.905, esp32s3 0.302-0.316) - the real 2026-11-07 regime shift (uno 0.473 -> 0.887, esp32s3 0.945 -> 0.416) is 10x outside the margin and still fires - replayed over the published history the stable window now reports 0% false alarms, while full-history fires concentrate on the genuine shifts Tests replay the real in-window Nightly values as a regression fixture.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2001.
What
The nightly cold-regression annotation warned whenever a board's
fbuild/PlatformIOcold ratio landed above the trailing 7-day median —true by construction ~half the time, since a median sits in the middle of
its distribution. This adds a relative margin:
Evidence
The 2026-11-12 nightly (run 43155336860,
5360eae2) fired on an 0.2%miss (uno 0.898 vs median 0.896) while fbuild was 3.3x faster than
PlatformIO on ESP32-S3 (202.8 s vs 671.8 s) and ~10% faster on the Uno.
Old rule measured by replaying all 161 published
benchmark-stats:history.jsonlrecords through the real
ratio_regressed_for_board:New rule on the same data:
The residual full-history fires are the genuine regime transitions
(2026-11-07: uno 0.473 -> 0.887, esp32s3 0.945 -> 0.416) — exactly what the
gate is for. Quiet-regime spread is ±1.5%, so 5% is ~4x the noise.
Tests
Three new tests in
build_comparison_tests.rs, TDD (the noise test failedagainst the old comparison first):
ratio_regression_ignores_nightly_noise_inside_margin— the real 18in-window Nightly values as a fixture: 0.898 no longer fires, a 6%
regression still does.
ratio_regression_fires_at_the_margin_boundary_only_past_it— strict>at exactly
median * 1.05.ratio_regression_catches_a_regime_shift— a 0.302-0.316 baseline stillfires at 0.95.
Existing
ratio_regression_uses_seven_day_median/..._uses_only_matching_board_historypass unchanged (their deltas are6-14%, above the margin).
cargo fmt --all -- --checkclean;clippy -p fbuild-bench-fastled-examples --all-targets --no-deps -- -D warnings -A unfulfilled_lint_expectationsclean.Docs
New "Cold-regression gate" section in
bench/fastled-examples/README.mdrecording the rule, the measured noise band and the replay numbers, so the
next person does not have to re-derive why the margin exists.
Deliberately out of scope
The trailing median moves with the data, so a slow multi-week drift never
trips it. That needs a fixed reference and is a separate design change
(noted in the issue).