Skip to content

fix(bench): compare cold ratios only within one measurement environment - #2004

Merged
zackees merged 1 commit into
mainfrom
agent/bench-gate-env
Nov 12, 2026
Merged

zackees merged 1 commit into
mainfrom
agent/bench-gate-env

Conversation

@zackees

@zackees zackees commented Nov 12, 2026

Copy link
Copy Markdown
Member

Closes #2003.

Why

The nightly's cold-regression gate (#2001/#2002) compares each board's fbuild/PlatformIO cold ratio against the trailing 7-day median of benchmark-stats:history.jsonl — but nothing in that history says how a row was measured. A ratio is as much a property of the harness as of fbuild.

The 2026-11-07 excursion that #2001 was filed about is exactly that: the first nightly after 0663b5da / 4179d196 / 8943ee99 made cold include first-time installs, every tool on both boards rose together:

nightly uno Arduino CLI uno PIO uno fbuild S3 Arduino CLI S3 PIO S3 fbuild
2026-11-06T09:35 1.0 s 0.9 s 0.45 s 13.2 s 264.0 s* 214.1 s
2026-11-07T09:26 59.4 s 61.4 s 54.4 s 1598.4 s 650.0 s 270.7 s

A 120x jump in Arduino CLI's own ESP32-S3 cold build cannot be an fbuild regression. It is the install time moving inside the timed window.

What changed

  • Every new history.jsonl row carries env = { network_mbit, cold_trial }, where cold_trial is COLD_TRIAL_SIGNATURE ("first-time-install/1").
  • ratio_regressed / ratio_regressed_for_board skip rows whose env differs, and skip rows with no env (all pre-change rows). The gate therefore never compares across a measurement boundary.
  • The warning text now says same-environment 7-day median.
  • Bumping COLD_TRIAL_SIGNATURE is the documented way to change cold-trial semantics: the gate goes quiet until the window refills instead of warning for a week against an incomparable baseline.
  • REGRESSION_MARGIN's doc comment corrected: the 2026-11-07 shift is a semantics change, not an fbuild slowdown.
  • bench/fastled-examples/README.md: new "Only rows measured the same way are compared" section with the table and the bump rule; the drift note now cites the measured weekly medians.

Tests (real output)

TDD: the new tests were written first and failed against the old code (cannot find type RunEnvironment, this function takes 3 arguments but 4 were supplied), then the implementation was added. One real bug was caught by the tests: ratio_regressed_for_board rebuilt each row as {ts, fbuild_vs_platformio_cold} and dropped env, which made every comparison vacuous.

soldr cargo test --bin bench-build-comparison
test result: ok. 60 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

(58 -> 60: ratio_regression_ignores_rows_from_another_environment and history_rows_record_the_measurement_environment are new; the three margin tests from #2002 are kept, adapted, and ratio_regression_catches_a_regime_shift became ratio_regression_catches_a_step_change_inside_one_environment — a step change inside one environment.)

cargo fmt --check clean; clippy -p fbuild-bench-fastled-examples --all-targets --no-deps -D warnings -A unfulfilled_lint_expectations clean (exit 0).

Effect on the published history

Replaying benchmark-stats:history.jsonl (161 rows) through the shipped rule: every existing row has no env, so the gate has 0 comparable rows and stays silent until the 7-day window refills — correct, since no existing row records its environment. Had the 2026-11-07+ rows been back-stamped, the medians would be uno 0.897 / esp32s3 0.308 with the newest run at 0.898 / 0.302 and nothing would fire.

Not done

  • No new bench matrix run: the push and scheduled Build Comparison Benchmark jobs on 66b64fd1 are already in flight and will exercise this gate for real (they are the reason this PR is a harness change and not a measurement claim).
  • No drift (trend) gate: weekly medians over the 7 weeks to 2026-11-12 stay inside ±5% (uno 0.447-0.471 for cycles 40-45, esp32s3 0.583-0.607), so a slower-moving detector has nothing to catch yet; noted as future work in the README instead.

The nightly's cold-regression gate compared each board's fbuild/PlatformIO
cold ratio against a trailing 7-day median taken from
`benchmark-stats:history.jsonl` no matter how those rows were measured. A
ratio is a property of the harness as much as of fbuild: on 2026-11-07, the
first nightly after `0663b5da`/`4179d196`/`8943ee99` made cold include
first-time installs, every tool on both boards rose together (uno Arduino CLI
1.0 s -> 59.4 s, ESP32-S3 Arduino CLI 13.2 s -> 1598.4 s, uno fbuild/PIO
0.473 -> 0.887). That was a change in what "cold" measures, not an fbuild
regression, yet the gate reported it as one and would have kept doing so for
the whole window.

History rows now record the environment they were measured in
(`env.network_mbit` + `env.cold_trial` = `COLD_TRIAL_SIGNATURE`), and the gate
skips rows whose environment differs — including pre-change rows with no
`env` at all. Bumping `COLD_TRIAL_SIGNATURE` is the documented way to change
cold-trial semantics: the gate then stays quiet until the window refills
instead of warning for a week against an incomparable baseline.

#2003
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@zackees
zackees added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 3caf6a9 Nov 12, 2026
29 checks passed
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

bench: cold-regression gate compares rows measured under different cold-trial semantics

1 participant