Skip to content

fix(bench): require a 5% margin before reporting a cold-build regression - #2002

Merged
zackees merged 1 commit into
mainfrom
agent/bench-regression-margin
Nov 12, 2026
Merged

zackees merged 1 commit into
mainfrom
agent/bench-regression-margin

Conversation

@zackees

@zackees zackees commented Nov 12, 2026

Copy link
Copy Markdown
Member

Fixes #2001.

What

The nightly cold-regression annotation warned whenever a board's
fbuild/PlatformIO cold ratio landed above the trailing 7-day median —
true by construction ~half the time, since a median sits in the middle of
its distribution. This adds a relative margin:

current_ratio > baseline * (1.0 + REGRESSION_MARGIN)   // REGRESSION_MARGIN = 0.05

Evidence

The 2026-11-12 nightly (run 43155336860, 5360eae2) fired on an 0.2%
miss (uno 0.898 vs median 0.896) while fbuild was 3.3x faster than
PlatformIO on ESP32-S3 (202.8 s vs 671.8 s) and ~10% faster on the Uno.

Old rule measured by replaying all 161 published benchmark-stats:history.jsonl
records through the real ratio_regressed_for_board:

window uno fires esp32s3 fires
full history 42/79 = 53% 28/80 = 35%
2026-11-07 onward 8/15 = 53% 2/15 = 13%
2026-11-08..12 (quiet regime) 6/12 = 50% 4/12 = 33%

New rule on the same data:

window uno fires esp32s3 fires
full history 23/79 = 29% 7/80 = 9%
2026-11-07 onward 0/15 0/15
2026-11-08..12 (quiet regime) 0/12 0/12

The residual full-history fires are the genuine regime transitions
(2026-11-07: uno 0.473 -> 0.887, esp32s3 0.945 -> 0.416) — exactly what the
gate is for. Quiet-regime spread is ±1.5%, so 5% is ~4x the noise.

Tests

Three new tests in build_comparison_tests.rs, TDD (the noise test failed
against the old comparison first):

  • ratio_regression_ignores_nightly_noise_inside_margin — the real 18
    in-window Nightly values as a fixture: 0.898 no longer fires, a 6%
    regression still does.
  • ratio_regression_fires_at_the_margin_boundary_only_past_it — strict >
    at exactly median * 1.05.
  • ratio_regression_catches_a_regime_shift — a 0.302-0.316 baseline still
    fires at 0.95.

Existing ratio_regression_uses_seven_day_median /
..._uses_only_matching_board_history pass unchanged (their deltas are
6-14%, above the margin).

$ soldr cargo test -p fbuild-bench-fastled-examples --bin bench-build-comparison
test result: ok. 58 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cargo fmt --all -- --check clean; clippy -p fbuild-bench-fastled-examples --all-targets --no-deps -- -D warnings -A unfulfilled_lint_expectations clean.

Docs

New "Cold-regression gate" section in bench/fastled-examples/README.md
recording the rule, the measured noise band and the replay numbers, so the
next person does not have to re-derive why the margin exists.

Deliberately out of scope

The trailing median moves with the data, so a slow multi-week drift never
trips it. That needs a fixed reference and is a separate design change
(noted in the issue).

The nightly gate compared each board's fbuild/PlatformIO cold ratio against
the trailing 7-day median and warned on any excess. A median sits in the
middle of its distribution, so the rule fired by construction roughly half
the time: replaying benchmark-stats:history.jsonl gives 53% (uno) and 35%
(esp32s3) over the full history, and 50%/33% across the quiet 2026-11-08..12
nights.

The 2026-11-12 run (5360eae) hit exactly that: uno 0.898 vs a 0.896 median,
a 0.2% miss, on a night when fbuild was 3.3x faster than PlatformIO on
ESP32-S3 and ~10% faster on the Uno. A gate that cries wolf on noise trains
everyone to ignore the annotation.

Require `current > median * (1 + REGRESSION_MARGIN)` with a 5% margin:
- measured night-to-night spread in the current regime is +-1.5%
  (uno 0.879-0.905, esp32s3 0.302-0.316)
- the real 2026-11-07 regime shift (uno 0.473 -> 0.887, esp32s3 0.945 ->
  0.416) is 10x outside the margin and still fires
- replayed over the published history the stable window now reports 0%
  false alarms, while full-history fires concentrate on the genuine shifts

Tests replay the real in-window Nightly values as a regression fixture.
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@zackees
zackees added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 66b64fd Nov 12, 2026
29 checks passed
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

bench: cold-regression gate fires on run-to-run noise (median compare has no margin)

1 participant