Skip to content

ci(bench): give the nightly its own concurrency slot, union history at publish - #2005

Merged
zackees merged 1 commit into
mainfrom
agent/bench-nightly-concurrency
Nov 12, 2026
Merged

zackees merged 1 commit into
mainfrom
agent/bench-nightly-concurrency

Conversation

@zackees

@zackees zackees commented Nov 12, 2026

Copy link
Copy Markdown
Member

ci(bench): give the nightly its own concurrency slot, union history at publish

The nightly Build Comparison Benchmark is silently starved on any day that a
merge lands while it is queued. concurrency: used a ref-only group for every
trigger, and GitHub keeps at most ONE pending run per group: a newer run
replaces a pending one, and cancel-in-progress: false only protects the run
that is already executing.

Evidence (real runs, gh run list --workflow=benchmark-build-comparison.yml):

run event created ended
43185178196 push (merge of #2002) 08:42:38Z success 11:10:53Z
43190818367 schedule (nightly) 09:21:59Z cancelled 10:28:43Z
43200708045 push (merge of #2004) 10:28:41Z in progress

The nightly queued behind the 08:42 push run, and died two seconds after the
10:28 push run was created. That is a hole in benchmark-stats/history.jsonl
that can never be backfilled.

Fix, in three parts:

  • benchmark-build-comparison.yml scopes the group by event -- push runs keep
    one slot per ref (a newer merge supersedes a queued measurement of the older
    sha), the nightly owns benchmark-build-comparison-nightly.
  • ci/check_workflow_concurrency.py now enforces this for every workflow: a
    schedule:-triggered workflow either declares no concurrency: block
    (nothing can evict it), a group with github.event_name, or a per-run group
    via github.run_id (esp32s3-size-parity.yml). Its docstring already
    described the pending-run rule; it just never applied outside PR workflows.
  • Overlap is the consequence, so the publisher no longer overwrites
    history.jsonl. Each run snapshots that file when its measurement STARTS,
    and the two publishers force-push the same branch, so an older snapshot
    landing last would revert the newer run's row. ci/merge_bench_history.py
    unions the published copy with the run's fresh rows: identity (ts, sha),
    chronological output with the newest line last (matching write_history),
    the same 365-line bound trimming the OLDEST row, and an unparseable line is
    passed through rather than deleted.

Real-data check (the published 162-row history, a rival push row vs a fresh
snapshot that predates it):

history: published=163 fresh=163 merged=164
merged rows: 164 | rival row kept: True | my row kept: True
newest last: 2026-11-12T10:50:00Z | chronological: True
plain overwrite would keep rival row: False   <-- the bug being removed

Tests (all real output):

uv run --with pyyaml --no-project python -m unittest ci.test_local_suites \
  ci.test_gate_script_coverage ci.test_workflow_concurrency \
  ci.test_merge_bench_history ci.test_benchmark_publish
Ran 57 tests ... OK

uv run --no-project python ci/check_python_complexity.py  -> All checks passed!
uv run --no-project python ci/render_workflows.py --check -> exit 0
bash -n over every `run:` block of the modified workflow -> all parse

Red first: the new eviction rule failed against the untouched repo
(benchmark-build-comparison.yml: the scheduled run shares concurrency group 'benchmark-build-comparison-${{ github.ref }}' with the other triggers ...), and
the first draft of that rule also flagged esp32s3-size-parity.yml, which is
already safe via github.run_id -- hence the per-run exemption and its test.

Not done here: no bench matrix run (the stale, redundant measurement is what
this fixes) and no change to the 2.5 h runtime or the nightly's 09:17Z slot.

…t publish

The nightly Build Comparison Benchmark is silently starved on any day that a
merge lands while it is queued. `concurrency:` used a ref-only group for every
trigger, and GitHub keeps at most ONE *pending* run per group: a newer run
replaces a pending one, and `cancel-in-progress: false` only protects the run
that is already executing.

Evidence (real runs, `gh run list --workflow=benchmark-build-comparison.yml`):

| run | event | created | ended |
|---|---|---|---|
| 43185178196 | push (merge of #2002) | 08:42:38Z | success 11:10:53Z |
| 43190818367 | schedule (nightly) | 09:21:59Z | **cancelled 10:28:43Z** |
| 43200708045 | push (merge of #2004) | 10:28:41Z | in progress |

The nightly queued behind the 08:42 push run, and died two seconds after the
10:28 push run was created. That is a hole in `benchmark-stats/history.jsonl`
that can never be backfilled.

Fix, in three parts:

* `benchmark-build-comparison.yml` scopes the group by event -- push runs keep
  one slot per ref (a newer merge supersedes a queued measurement of the older
  sha), the nightly owns `benchmark-build-comparison-nightly`.
* `ci/check_workflow_concurrency.py` now enforces this for every workflow: a
  `schedule:`-triggered workflow either declares no `concurrency:` block
  (nothing can evict it), a group with `github.event_name`, or a per-run group
  via `github.run_id` (`esp32s3-size-parity.yml`). Its docstring already
  described the pending-run rule; it just never applied outside PR workflows.
* Overlap is the consequence, so the publisher no longer overwrites
  `history.jsonl`. Each run snapshots that file when its measurement STARTS,
  and the two publishers force-push the same branch, so an older snapshot
  landing last would revert the newer run's row. `ci/merge_bench_history.py`
  unions the published copy with the run's fresh rows: identity `(ts, sha)`,
  chronological output with the newest line last (matching `write_history`),
  the same 365-line bound trimming the OLDEST row, and an unparseable line is
  passed through rather than deleted.

Real-data check (the published 162-row history, a rival push row vs a fresh
snapshot that predates it):

```
history: published=163 fresh=163 merged=164
merged rows: 164 | rival row kept: True | my row kept: True
newest last: 2026-11-12T10:50:00Z | chronological: True
plain overwrite would keep rival row: False   <-- the bug being removed
```

Tests (all real output):

```
uv run --with pyyaml --no-project python -m unittest ci.test_local_suites \
  ci.test_gate_script_coverage ci.test_workflow_concurrency \
  ci.test_merge_bench_history ci.test_benchmark_publish
Ran 57 tests ... OK

uv run --no-project python ci/check_python_complexity.py  -> All checks passed!
uv run --no-project python ci/render_workflows.py --check -> exit 0
bash -n over every `run:` block of the modified workflow -> all parse
```

Red first: the new eviction rule failed against the untouched repo
(`benchmark-build-comparison.yml: the scheduled run shares concurrency group
'benchmark-build-comparison-${{ github.ref }}' with the other triggers ...`), and
the first draft of that rule also flagged `esp32s3-size-parity.yml`, which is
already safe via `github.run_id` -- hence the per-run exemption and its test.

Not done here: no bench matrix run (the stale, redundant measurement is what
this fixes) and no change to the 2.5 h runtime or the nightly's 09:17Z slot.
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

1 participant