Skip to content

Retry setup-python on a transient download failure - #1803

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/setup-python-retry
Oct 29, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/setup-python-retry

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

On 2026-10-28 the e2e leg e2e (ubuntu-latest, e2e_vendor_composer_build e2e_redirect_composer_build e2e_redirect_uv_build …) of run 40771482790 (attempt 1, job 123111232415, Depot runner) failed in Setup Python:

21:26:46 Version 3.12.10 was not found in the local cache
21:30:14 ##[error]getaddrinfo EAI_AGAIN raw.githubusercontent.com

The step didn't finish until 21:45:25, so it held the leg for 19 min before going red. Re-running the job passed, so this was a DNS blip, not a real failure. A healthy Setup Python takes 0–6 s (merge_group run 40782886803). On a PR this costs a re-run. In the merge queue the same blip evicts the entry and every entry behind it rebuilds.

Frequency is low: this is the only setup-* step failure among failed CI jobs since 2026-10-22. It is still the only infra flake in the last ~10 h of CI that can be fixed in the repo (the other one was a Depot runner loss, which the merge-queue fail-fast job already recognises). Python-using e2e legs run on every PR and in every merge_group run, and #1740 (yarn EAI_AGAIN) fixed the same class of failure.

Root cause

actions/setup-python downloads 3.12.10 on runners that don't cache it. It fetches the versions manifest from raw.githubusercontent.com, which has no retry for a DNS failure, and the step has no time limit.

Fix

  • In ci.yml, both setup-python calls (e2e and hosted-e2e) now run as (attempt 1) with timeout-minutes: 4 and continue-on-error: true, then (attempt 2) with the same pin and inputs, which runs only when attempt 1's outcome isn't success. Attempt 2 doesn't tolerate failure, so a real outage still fails the leg. This is the same pattern as .github/actions/download-artifact.
  • The ci.yml contract test (scripts/tests/test_merge_queue_fail_fast.py::test_no_job_tolerates_failure) used to let only capped cache restores carry continue-on-error. It now also accepts a capped (attempt 1) step, but only when the very next step is an (attempt 2) of the same uses:, gated on steps.<id>.outcome != 'success' and without continue-on-error. A tolerated step can no longer hide a failure from ci-ok or the fail-fast cancel.
  • hosted-e2e's step was named "Setup Python + uv", but uv is installed by the next step, so it is now "Setup Python (attempt 1/2)". Nothing references the old name.

No tests were removed or moved, and required-check names and ci-ok are unchanged.

Proof

  • python3 -B -m unittest discover -s scripts/tests passes: 376 tests, 6 skipped. Before the test change, the contract test failed on the new attempt-1 step, so it does check this.
  • The YAML parses. Step ids are unique in each job, and both jobs have the expected if, timeout-minutes and continue-on-error values.
  • I couldn't run actionlint or zizmor because they aren't installed in this session.
  • The DNS failure can't be reproduced locally. The retry path will run the next time setup-python's download fails on a runner.

https://claude.ai/code/session_011f9aeRTxATcS2gdrSf6uDJ


Generated by Claude Code

On 2026-10-28 an e2e leg (run 40771482790, job 123111232415) failed in
Setup Python: the 3.12.10 download died on `getaddrinfo EAI_AGAIN
raw.githubusercontent.com` and the step held the runner for 19 min
before going red. A healthy setup takes ~6 s. In the merge queue the
same blip evicts the entry and costs a full rebuild.

Cap each setup-python call in ci.yml (e2e and hosted-e2e) at 4 min and
retry it once when the first attempt fails or times out. The second
attempt does not tolerate failure, so a real outage still fails the
leg.

The ci.yml contract test only let capped cache restores carry
continue-on-error. Also allow a capped "(attempt 1)" step whose very
next step is an "(attempt 2)" of the same action, gated on the first
attempt's outcome and without continue-on-error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 29, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 54ee197. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 29, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: labeled Ready for review at 54ee197.

  • CI: 51 check runs: 32 success, 19 skipped, 0 failing on 54ee197; mergeable, no conflicts.
  • Cursor Bugbot: reviewed 54ee197, no issues found; no unresolved review threads.
  • No CHANGELOG.md change.
  • Slack announcement could not be sent in this run (no Slack send tool available); the next run will retry.

Generated by Claude Code

@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does: The two actions/setup-python steps in ci.yml (the e2e matrix and the hosted-e2e job) are each split into a 4-minute-capped first attempt with continue-on-error and a second, also capped, attempt without continue-on-error that runs only if the first failed. A DNS blip during the 3.12.10 download held a leg for 19 minutes and then failed it. The workflow contract test now allows a step-level continue-on-error only on capped cache restores, or on an "(attempt 1)" step that is immediately followed by a matching "(attempt 2)" with the same uses: and no tolerance.

Risk: low. This is CI-only. A real failure still fails the leg at attempt 2, so ci-ok can't go green on a broken setup. A gated-off attempt 1 can't trigger attempt 2 because attempt 2 repeats the gate.

Look here:

Verified: I read the full diff. The step ids don't collide within either job, and the old step name "Setup Python + uv" has no other references. CHANGELOG.md is untouched. CI: ci-ok and clippy green on 54ee197, all checks pass. Cursor Bugbot: success, no threads.

Changes I made: none.

Open questions: none.

I'm arming auto-merge, so approving sends it straight to the merge queue.


Generated by Claude Code

Merged via the queue into main with commit ce11f43 Oct 29, 2026
51 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/setup-python-retry branch October 29, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants