Skip to content

ci(bench): publish under a lease so overlapping runs cannot revert history - #2007

Merged
zackees merged 1 commit into
mainfrom
agent/bench-publish-race
Nov 12, 2026
Merged

zackees merged 1 commit into
mainfrom
agent/bench-publish-race

Conversation

@zackees

@zackees zackees commented Nov 12, 2026

Copy link
Copy Markdown
Member

What

benchmark-build-comparison.yml's publish step ended with

git -C "${publish_dir}" push --force "${repo_url}" benchmark-stats

Since #2005 the nightly and a push run own separate concurrency: groups, so
two publish jobs can overlap. ci/merge_bench_history.py only helps when the
published copy is read after the rival's push. When both publishes interleave
(both fetch, then both push), --force cannot fail and the loser's tree silently
reverts the winner's row — and history.jsonl is append-only, so that night is a
permanent hole. That is the exact loss #2005 set out to stop.

Reproduced, not inferred

Two real publishers against one real bare remote, seeded with the live published
history (162 rows), using the current workflow's bash with the fetches interleaved:

history: published=5 fresh=6 merged=6 -> pub-A/history.jsonl
history: published=5 fresh=6 merged=6 -> pub-B/history.jsonl
--- final rows on benchmark-stats ---
rows 6 ['runBBBB', 'seed0000', ... 'seed0004']
A kept: False | B kept: True

A published first; B, which had snapshotted the branch before A's push,
force-pushed its own tree and A's row was gone.

Fix

New ci/publish_bench_site.py (the workflow's inline bash becomes a call to it):

  • every attempt re-reads benchmark-stats first, so the union input is what
    is published now, not what was published when the run started;
  • the push is --force-with-lease=refs/heads/benchmark-stats:<sha-it-read> (an
    empty expected value means "the branch must not exist yet"), so a rival's push
    becomes a rejected push instead of a lost row;
  • a rejection re-assembles and repushes — idempotent, because the union
    de-duplicates on (ts, sha) — and after --max-attempts the step fails loudly
    instead of clobbering;
  • a non-lease failure (auth, network) fails immediately rather than burning the
    retry budget.

ci/ is imported as a namespace package in the sparse checkout, so the publisher
needs only its two modules checked out (listing the package marker would make
ci/gate_script_coverage.py read it as an invoked gate script).

Tests

ci/test_publish_bench_site.py — 6 tests driving real git repositories (a
bare origin plus two independent publish dirs), because the defect lives in
git's ref semantics:

  • sequential publishers keep both rows;
  • interleaved publishers keep both rows — RED under the old single-shot
    --force push (AssertionError: 'runBBBB' not found in ['seed0000', 'seed0001', 'seed0002', 'runAAAA']),
    GREEN with the lease + retry;
  • repeated rejections raise PublishError instead of clobbering;
  • an unchanged site does not push (remote tip unchanged);
  • a measurement-less run keeps the published report and still syncs the mirror;
  • the push is lease-protected (no plain --force in the module).

ci/test_benchmark_publish.py was updated to assert the wiring (the step runs
-m ci.publish_bench_site, the sparse checkout carries both modules, the mirror
is copied by the publisher, a no-op publish never pushes) instead of the removed
inline bash.

Gates run locally:

uv run --with pyyaml --no-project python -m unittest <all 30 DRIFT_UNITTEST_MODULES>
Ran 483 tests in 13.932s
OK

uv run --no-project --with ruff==0.12.12 ruff format --check ...   -> 4 files already formatted
ruff check --isolated --select E,F,W,I,UP,B ...                    -> All checks passed!
python ci/check_python_complexity.py                               -> All checks passed!
python ci/check_workflow_concurrency.py                            -> OK
python ci/find_direct_subprocess.py                                -> exit 0

End-to-end against a bare mirror seeded with the live 162-row benchmark-stats
history, running the wired command:

published benchmark-stats on attempt 1 (was 72d1d691...)
rows 163   fresh kept: True   index: <html>fresh report</html>   mirror entries: 8046

Refs #2006

…story

Since #2005 the nightly and a push run own separate concurrency slots, so two
`publish` jobs can run at the same time. The history union only helps when the
published copy is read after the rival's push; the publisher closed the window
with `git push --force`, which cannot fail, so a rival that landed between the
fetch and the push was silently reverted -- and `history.jsonl` is append-only,
so that measurement was a permanent hole.

Reproduced with two real publishers against one bare remote seeded with the
live 162-row history: A pushed its row, then B (which had snapshotted the
branch before A's push) force-pushed its own tree and the final history held
the seed rows plus B only.

`ci/publish_bench_site.py` now re-reads the branch on every attempt and pushes
with `--force-with-lease=refs/heads/benchmark-stats:<sha-it-read>`; a rejection
re-assembles and retries (idempotent: the union de-duplicates on `(ts, sha)`)
and exhausting the budget fails the step loudly. The workflow's inline bash
becomes a call to that module, which is covered by 6 tests driving real git
repositories (bare remote, two publish dirs); the interleaved test is red under
the old force push (`'runBBBB' not found in [... 'runAAAA']`).

Refs #2006
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

1 participant