Skip to content

ci(bench): never let an older measurement overwrite a newer site report - #2010

Merged
zackees merged 1 commit into
mainfrom
agent/bench-publish-order
Nov 12, 2026
Merged

zackees merged 1 commit into
mainfrom
agent/bench-publish-order

Conversation

@zackees

@zackees zackees commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

What

ci/publish_bench_site.py now recency-gates the measurement overlay on
metadata.generated_at, so a report that is not newer than the one already on
benchmark-stats can never replace it. Its history row is still unioned, and
site/mirror/** is still published.

Closes #2009.

Why

#2005 gave the nightly its own concurrency slot (so overlap is now possible
instead of starving the nightly), #2006/#2007 made the overlap safe on the
branch (re-read + --force-with-lease + (ts, sha) union), and #2008
serialised the Pages deployment. The remaining gap was the content of
the deployed tree: index.html, benchmark.svg and latest.json describe
exactly one run, and the publisher copied the whole measurements directory over
the published tree unconditionally.

The run that finishes last is then not necessarily the run that measured
last. The nightly starts at 09:17Z and takes ~2.5 h, so a merge landing at
10:00Z publishes its (newer) push measurement first, and the nightly can then
republish a report for an older sha: the site's latest.json then points at
a sha older than one already published, while history.jsonl correctly holds
both rows — the site contradicting its own history until the next measurement
lands.

Evidence (real git, no mocks)

Bare mirror seeded from the live branch (28d634d0, 164 rows, newest row
b7d1bdf0 @ 2026-11-12T13:48:03Z), publisher run for a report stamped
13:40:00Z:

Published report is newer than this run's; keeping the published site and unioning history only.
published benchmark-stats on attempt 1 (was 28d634d0...)
rows=165                     # the extra row is unioned, nothing lost
index=<!doctype html>        # live report preserved
latest_sha=b7d1bdf0          # live report preserved

Before the fix the same scenario left index.html = the older run's page
(new test RED: '<html>runNEW00</html>' != '<html>runOLD00</html>').

The inverse case still publishes: bumping the stamp to 20:00:00Z replaced the
site (index -> the new page, latest_sha -> the new sha) and the row count
stayed 165, so the union stays idempotent and the gate is not an always-skip.

Tests

  • 3 new tests in ci/test_publish_bench_site.py, driving real git repos:
    older-publishes-later keeps the newer report while keeping both rows;
    an undated published report never blocks a publish (old-schema branch cannot
    wedge the pipeline); a stale run still publishes the mirror (mirror is
    checkout content, never gated).
  • ci.test_publish_bench_site: 9 tests OK.
  • CI drift suite (the workflow's own unittest list): Ran 486 tests — OK
    (ruff==0.12.12 format/check clean; ci/check_python_complexity.py clean).
  • Workflow comment updated to record the third leg of the overlap chain.

Since #2005 the nightly owns its own workflow-level concurrency slot, so a
nightly and a push run can overlap. Unioning history.jsonl by (ts, sha) keeps
both rows, but the rest of the site tree (index.html, benchmark.svg,
latest.json) describes exactly ONE run -- and the run that finishes LAST is not
necessarily the run that measured LAST. The nightly starts at 09:17Z and takes
~2.5 h, so a merge landing mid-run publishes its (newer) push measurement
first, and the nightly then copies an older report over it: the site then
contradicts its own history until the next measurement lands.

Gate the measurement overlay on metadata.generated_at (the harness stamp
date -u +'%Y-%m-%dT%H:%M:%SZ'): a report that is not newer than the published
one is skipped with a log line, while

  * its history row is still unioned (the measurement is never lost), and
  * site/mirror/** is still published -- it is checkout content, always
    current, so it must never be gated on report recency.

Unknown or unparseable stamps on either side fall back to publishing, so an
older-schema branch cannot wedge the pipeline.

Reproduced on real git repos: publishing a 13:40Z report on top of a published
13:48:03Z one left index.html as the older run's page (RED); with the gate the
live report and latest.json survive and the branch gains the extra row (165).
The inverse case (newer stamp) still replaces the site, so the gate is not an
always-skip. Tests 9/9 in ci.test_publish_bench_site, CI drift suite 486 OK.

Closes #2009
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@zackees
zackees added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 35c68d0 Nov 12, 2026
29 checks passed
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

ci(bench): an older measurement can publish after a newer one and overwrite the site report

1 participant