Repository navigation
fix(bench): compare cold ratios only within one measurement environment - #2004
Merged
Merged
Conversation
The nightly's cold-regression gate compared each board's fbuild/PlatformIO cold ratio against a trailing 7-day median taken from `benchmark-stats:history.jsonl` no matter how those rows were measured. A ratio is a property of the harness as much as of fbuild: on 2026-11-07, the first nightly after `0663b5da`/`4179d196`/`8943ee99` made cold include first-time installs, every tool on both boards rose together (uno Arduino CLI 1.0 s -> 59.4 s, ESP32-S3 Arduino CLI 13.2 s -> 1598.4 s, uno fbuild/PIO 0.473 -> 0.887). That was a change in what "cold" measures, not an fbuild regression, yet the gate reported it as one and would have kept doing so for the whole window. History rows now record the environment they were measured in (`env.network_mbit` + `env.cold_trial` = `COLD_TRIAL_SIGNATURE`), and the gate skips rows whose environment differs — including pre-change rows with no `env` at all. Bumping `COLD_TRIAL_SIGNATURE` is the documented way to change cold-trial semantics: the gate then stays quiet until the window refills instead of warning for a week against an incomparable baseline. #2003
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2003.
Why
The nightly's cold-regression gate (#2001/#2002) compares each board's
fbuild/PlatformIOcold ratio against the trailing 7-day median ofbenchmark-stats:history.jsonl— but nothing in that history says how a row was measured. A ratio is as much a property of the harness as of fbuild.The 2026-11-07 excursion that #2001 was filed about is exactly that: the first nightly after
0663b5da/4179d196/8943ee99made cold include first-time installs, every tool on both boards rose together:A 120x jump in Arduino CLI's own ESP32-S3 cold build cannot be an fbuild regression. It is the install time moving inside the timed window.
What changed
history.jsonlrow carriesenv = { network_mbit, cold_trial }, wherecold_trialisCOLD_TRIAL_SIGNATURE("first-time-install/1").ratio_regressed/ratio_regressed_for_boardskip rows whoseenvdiffers, and skip rows with noenv(all pre-change rows). The gate therefore never compares across a measurement boundary.COLD_TRIAL_SIGNATUREis the documented way to change cold-trial semantics: the gate goes quiet until the window refills instead of warning for a week against an incomparable baseline.REGRESSION_MARGIN's doc comment corrected: the 2026-11-07 shift is a semantics change, not an fbuild slowdown.bench/fastled-examples/README.md: new "Only rows measured the same way are compared" section with the table and the bump rule; the drift note now cites the measured weekly medians.Tests (real output)
TDD: the new tests were written first and failed against the old code (
cannot find type RunEnvironment,this function takes 3 arguments but 4 were supplied), then the implementation was added. One real bug was caught by the tests:ratio_regressed_for_boardrebuilt each row as{ts, fbuild_vs_platformio_cold}and droppedenv, which made every comparison vacuous.(58 -> 60:
ratio_regression_ignores_rows_from_another_environmentandhistory_rows_record_the_measurement_environmentare new; the three margin tests from #2002 are kept, adapted, andratio_regression_catches_a_regime_shiftbecameratio_regression_catches_a_step_change_inside_one_environment— a step change inside one environment.)cargo fmt --checkclean;clippy -p fbuild-bench-fastled-examples --all-targets --no-deps -D warnings -A unfulfilled_lint_expectationsclean (exit 0).Effect on the published history
Replaying
benchmark-stats:history.jsonl(161 rows) through the shipped rule: every existing row has noenv, so the gate has 0 comparable rows and stays silent until the 7-day window refills — correct, since no existing row records its environment. Had the 2026-11-07+ rows been back-stamped, the medians would be uno 0.897 / esp32s3 0.308 with the newest run at 0.898 / 0.302 and nothing would fire.Not done
Build Comparison Benchmarkjobs on66b64fd1are already in flight and will exercise this gate for real (they are the reason this PR is a harness change and not a measurement claim).uno0.447-0.471 for cycles 40-45,esp32s30.583-0.607), so a slower-moving detector has nothing to catch yet; noted as future work in the README instead.