Repository navigation
ci(bench): give the nightly its own concurrency slot, union history at publish - #2005
Merged
Merged
Conversation
…t publish The nightly Build Comparison Benchmark is silently starved on any day that a merge lands while it is queued. `concurrency:` used a ref-only group for every trigger, and GitHub keeps at most ONE *pending* run per group: a newer run replaces a pending one, and `cancel-in-progress: false` only protects the run that is already executing. Evidence (real runs, `gh run list --workflow=benchmark-build-comparison.yml`): | run | event | created | ended | |---|---|---|---| | 43185178196 | push (merge of #2002) | 08:42:38Z | success 11:10:53Z | | 43190818367 | schedule (nightly) | 09:21:59Z | **cancelled 10:28:43Z** | | 43200708045 | push (merge of #2004) | 10:28:41Z | in progress | The nightly queued behind the 08:42 push run, and died two seconds after the 10:28 push run was created. That is a hole in `benchmark-stats/history.jsonl` that can never be backfilled. Fix, in three parts: * `benchmark-build-comparison.yml` scopes the group by event -- push runs keep one slot per ref (a newer merge supersedes a queued measurement of the older sha), the nightly owns `benchmark-build-comparison-nightly`. * `ci/check_workflow_concurrency.py` now enforces this for every workflow: a `schedule:`-triggered workflow either declares no `concurrency:` block (nothing can evict it), a group with `github.event_name`, or a per-run group via `github.run_id` (`esp32s3-size-parity.yml`). Its docstring already described the pending-run rule; it just never applied outside PR workflows. * Overlap is the consequence, so the publisher no longer overwrites `history.jsonl`. Each run snapshots that file when its measurement STARTS, and the two publishers force-push the same branch, so an older snapshot landing last would revert the newer run's row. `ci/merge_bench_history.py` unions the published copy with the run's fresh rows: identity `(ts, sha)`, chronological output with the newest line last (matching `write_history`), the same 365-line bound trimming the OLDEST row, and an unparseable line is passed through rather than deleted. Real-data check (the published 162-row history, a rival push row vs a fresh snapshot that predates it): ``` history: published=163 fresh=163 merged=164 merged rows: 164 | rival row kept: True | my row kept: True newest last: 2026-11-12T10:50:00Z | chronological: True plain overwrite would keep rival row: False <-- the bug being removed ``` Tests (all real output): ``` uv run --with pyyaml --no-project python -m unittest ci.test_local_suites \ ci.test_gate_script_coverage ci.test_workflow_concurrency \ ci.test_merge_bench_history ci.test_benchmark_publish Ran 57 tests ... OK uv run --no-project python ci/check_python_complexity.py -> All checks passed! uv run --no-project python ci/render_workflows.py --check -> exit 0 bash -n over every `run:` block of the modified workflow -> all parse ``` Red first: the new eviction rule failed against the untouched repo (`benchmark-build-comparison.yml: the scheduled run shares concurrency group 'benchmark-build-comparison-${{ github.ref }}' with the other triggers ...`), and the first draft of that rule also flagged `esp32s3-size-parity.yml`, which is already safe via `github.run_id` -- hence the per-run exemption and its test. Not done here: no bench matrix run (the stale, redundant measurement is what this fixes) and no change to the 2.5 h runtime or the nightly's 09:17Z slot.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
This was referenced Nov 12, 2026
Merged
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci(bench): give the nightly its own concurrency slot, union history at publish
The nightly Build Comparison Benchmark is silently starved on any day that a
merge lands while it is queued.
concurrency:used a ref-only group for everytrigger, and GitHub keeps at most ONE pending run per group: a newer run
replaces a pending one, and
cancel-in-progress: falseonly protects the runthat is already executing.
Evidence (real runs,
gh run list --workflow=benchmark-build-comparison.yml):The nightly queued behind the 08:42 push run, and died two seconds after the
10:28 push run was created. That is a hole in
benchmark-stats/history.jsonlthat can never be backfilled.
Fix, in three parts:
benchmark-build-comparison.ymlscopes the group by event -- push runs keepone slot per ref (a newer merge supersedes a queued measurement of the older
sha), the nightly owns
benchmark-build-comparison-nightly.ci/check_workflow_concurrency.pynow enforces this for every workflow: aschedule:-triggered workflow either declares noconcurrency:block(nothing can evict it), a group with
github.event_name, or a per-run groupvia
github.run_id(esp32s3-size-parity.yml). Its docstring alreadydescribed the pending-run rule; it just never applied outside PR workflows.
history.jsonl. Each run snapshots that file when its measurement STARTS,and the two publishers force-push the same branch, so an older snapshot
landing last would revert the newer run's row.
ci/merge_bench_history.pyunions the published copy with the run's fresh rows: identity
(ts, sha),chronological output with the newest line last (matching
write_history),the same 365-line bound trimming the OLDEST row, and an unparseable line is
passed through rather than deleted.
Real-data check (the published 162-row history, a rival push row vs a fresh
snapshot that predates it):
Tests (all real output):
Red first: the new eviction rule failed against the untouched repo
(
benchmark-build-comparison.yml: the scheduled run shares concurrency group 'benchmark-build-comparison-${{ github.ref }}' with the other triggers ...), andthe first draft of that rule also flagged
esp32s3-size-parity.yml, which isalready safe via
github.run_id-- hence the per-run exemption and its test.Not done here: no bench matrix run (the stale, redundant measurement is what
this fixes) and no change to the 2.5 h runtime or the nightly's 09:17Z slot.