Skip to content

ci: repair runner setup and exact-base comparisons - #579

Merged
Eduardo Villalpando Mello (edvilme) merged 2 commits into
mainfrom
stellahuang95-ci-infrastructure-repair
Nov 16, 2026
Merged

Eduardo Villalpando Mello (edvilme) merged 2 commits into
mainfrom
stellahuang95-ci-infrastructure-repair

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Provision Linux Miniconda explicitly instead of assuming $CONDA is preinstalled, without creating or activating an extra environment. Repair both PR workflows and baseline publishers.
  • Replace retired macOS 14 runners with supported macOS 15 ARM64 runners, and run x86_64 workloads on macos-15-intel. Match Conda installers to native architecture and retain both architectures' functional/build coverage.
  • Measure the exact PR base afresh. Coverage uses separate clean base/head jobs because Unix Conda CI tests create global environments and install Mamba into base. Performance builds each revision and harness in its own worktree, then measures both on the same runner against the same discovery workspace and fixtures.
  • Validate actual source SHAs, run-attempt-qualified coverage artifacts, matching nonempty measurement contexts, and successful test/proof/extraction outcomes before invoking the unchanged comparators.
  • Address review feedback by extracting argument-driven staging/capture/comparison into scripts/quality_measurement.py and adding a distinct native ARM64 lane to both performance workflows while retaining Intel.

Root cause and impact

Before this repair, Linux jobs failed before testing because Ubuntu 26.04 no longer supplied the Conda installation assumed by the workflows. The cancelled macOS jobs never received a runner: their macOS 14 configuration retired on November 2.

The original Windows numerical coverage failure compared Rust 1.99.1/LLVM 23.1.1 with Rust 1.100.0/LLVM 23.1.5 and different runner images. With the same 96 Rust source files, LCOV denominators changed from LF 23,480/FNF 2,231 to LF 23,288/FNF 2,254. This demonstrates measurement-environment drift, not that the compiler alone explains every changed hit.

The original macOS startup-P95 outlier is not treated as a proven product defect. Its one unchanged-head rerun passed the timing budgets but failed inventory matching: 12 environments versus the historical baseline's 11. Fresh comparisons address that comparability problem without discarding slow samples or normalizing inventories away.

Preserved invariants and review follow-up

  • No Rust/product, regression-budget, or snapshot-schema changes. The 0.01 percentage-point coverage tolerance, all performance budgets, required samples, and inventory checks remain intact.
  • Native ARM64 macOS coverage retains its default-feature workload and same-runner exact-base comparison. Installed-manager fixtures, environments without Python, subprocess proof, and fail-closed reporting remain required.
  • The extracted helper stages only the tested workflow's two immutable script blobs plus a revision marker outside the discovery tree before coverage checks out the measured source. An older base need not contain the new helper. helper_revision and the measured source SHA are validated separately; head Rust source/binaries are never substituted for the base.
  • Coverage and performance share common capture/provenance logic but retain their different required tools and fixtures. Numerical gates remain in the unchanged quality_snapshot.py.
  • Native ARM64 performance has its own artifacts/comment lane and freshly measured exact base. It uses the existing macOS budgets; Intel remains included.
  • Historical baseline workflows remain publishers for inspection. PR gates no longer depend on their older artifacts. Fresh comparisons add a real base measurement per platform.
  • Separate from ci: pin cargo-llvm-cov installer to release SHA #578: its branch and installer-pinning changes are untouched. No merge is requested.

Current validation

Current head: f10594fd644f112fa609dce548feab29d0dd4c92, a new locally GPG-signed commit preserving parent 31901b8. Both local GnuPG verification and GitHub's OpenPGP verification succeed. The trusted-team-key signing policy recognizes the signature; VS Code PR Check now passes with the required collaborator approval.

Actual measurements use exact base 02dd29e49e3c71d4ec06e1072d6331494a12e01d and tested merge d6d49a1ad7080a4e9bcda6259cd60a11fa3ab384 containing the current head.

  • Local Python suite: 116 tests, one skipped. The parent independently ran all 17 quality-workflow tests from the immutable signed revision. Tests exercise the real CLI and reject wrong revisions, missing staged helpers/dependencies, mismatched contexts, failed execution/extraction, and accidental reuse of head as base.
  • actionlint passes all three follow-up workflows; the original seven-workflow repair was also linted.
  • Refactored Linux/Windows coverage passes. Downloaded artifacts verify the old-base/new-helper case, matching contexts, and required zero-idle/positive-info handler/transport/writer witnesses for both revisions. Line deltas are Linux +0.004445pp and Windows -0.004294pp; function deltas are zero, within unchanged budgets.
  • All four paired performance lanes pass all 12 gates with all nine distributions containing ten samples each. Base/head environment inventories match: Linux 5/5, Windows 8/8, Intel macOS 10/10, ARM64 macOS 12/12; managers are 1/1 throughout. ARM64 startup P95 is 73ms base versus 65ms head.
  • The updated performance publisher passes on all four platforms. Its ARM64 VM independently has 11 environments/1 manager; that legitimate cross-run difference is retained, not compared against the paired workload or normalized away.

Remaining CI blockers

The review changes are pushed and all four reviewer threads have replies and are resolved, but the PR is not fully green:

  • Native ARM64 coverage passes functional tests, isolated proof, and reporting but fails the raw line gate by -0.031pp. Exactly one unchanged diagnostic rerun also failed: head LH/LF 18,146/22,512 versus base 18,153/22,512, with functions identical at 1,826/2,205. The seven-hit discrepancy moved between unchanged source paths across attempts; the second attempt's differing output.rs summaries have identical positive DA line sets. Both failed artifacts are retained. This is unresolved runtime/summary-hit variability, not a missing helper or removed test. No further rerun or tolerance change was made.
  • Intel macOS functional tests fail verify_validity_of_discovered_envs at ci_test.rs:629: full discovery includes one additional hosted-toolcache x64 Python symlink that direct try_from does not return. Interpreter identity/version/prefix/manager/architecture agree. Existing CI normalization excludes ARM64 toolcache aliases but not x64. The initial repair passed on the same image/Python version; why the extra alias appears in this run remains unproven. This follow-up changes neither pr-check.yml nor Rust and that job does not invoke the helper. No product-locator workaround or blanket retry was made.

These remaining reliability/alias-causality issues are reported explicitly rather than weakening gates or silently expanding the maintenance-feedback fix.

Restore explicit Linux Conda provisioning and supported native macOS runners.
Measure exact base/head revisions with isolated coverage fixtures and shared-runner
performance fixtures, validating source provenance, measurement context, and
execution outcomes before the unchanged quality gates.

Preserve ARM64 coverage, subprocess proof, inventories, and all regression budgets.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 1ms 1ms +0ms +0.0% >5ms and >100% ➖
Server startup P95 1ms 1ms +0ms +0.0% >50ms and >200% ➖
Discovery duration P50 57ms 56ms +1ms +1.8% >25ms and >30% 🔺
Discovery duration P95 58ms 58ms +0ms +0.0% >50ms and >50% ➖
Startup-to-first environment P50 12ms 11ms +1ms +9.1% >20ms and >100% 🔺
Startup-to-first environment P95 14ms 12ms +2ms +16.7% >25ms and >100% 🔺
Cold discovery duration P50 132ms 135ms -3ms -2.2% >100ms and >50% ✅
Refresh round-trip P50 57ms 56ms +1ms +1.8% >25ms and >30% 🔺
Refresh round-trip P95 58ms 58ms +0ms +0.0% >50ms and >50% ➖
Request-to-first environment P50 10ms 10ms +0ms +0.0% >20ms and >100% ➖
Request-to-first environment P95 12ms 11ms +1ms +9.1% >25ms and >100% 🔺
Cold refresh round-trip P50 132ms 136ms -4ms -2.9% >100ms and >50% ✅
Workload PR Baseline
Environments 5 5
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 85.451% 85.447% +0.004pp
Functions 88.022% 88.022% +0.000pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 9ms 10ms -1ms -10.0% >10ms and >50% ✅
Server startup P95 13ms 13ms +0ms +0.0% >50ms and >100% ➖
Discovery duration P50 156ms 159ms -3ms -1.9% >150ms and >50% ✅
Discovery duration P95 163ms 168ms -5ms -3.0% >250ms and >100% ✅
Startup-to-first environment P50 21ms 24ms -3ms -12.5% >25ms and >50% ✅
Startup-to-first environment P95 26ms 31ms -5ms -16.1% >100ms and >100% ✅
Cold discovery duration P50 157ms 155ms +2ms +1.3% >150ms and >50% 🔺
Refresh round-trip P50 156ms 160ms -4ms -2.5% >150ms and >50% ✅
Refresh round-trip P95 163ms 168ms -5ms -3.0% >250ms and >100% ✅
Request-to-first environment P50 11ms 14ms -3ms -21.4% >25ms and >50% ✅
Request-to-first environment P95 16ms 20ms -4ms -20.0% >100ms and >100% ✅
Cold refresh round-trip P50 158ms 156ms +2ms +1.3% >150ms and >50% 🔺
Workload PR Baseline
Environments 8 8
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 82.961% 82.965% -0.004pp
Functions 85.359% 85.359% +0.000pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 48ms 55ms -7ms -12.7% >100ms and >50% ✅
Server startup P95 63ms 131ms -68ms -51.9% >750ms and >100% ✅
Discovery duration P50 83ms 87ms -4ms -4.6% >100ms and >50% ✅
Discovery duration P95 115ms 106ms +9ms +8.5% >300ms and >100% 🔺
Startup-to-first environment P50 80ms 87ms -7ms -8.0% >150ms and >50% ✅
Startup-to-first environment P95 139ms 105ms +34ms +32.4% >250ms and >100% 🔺
Cold discovery duration P50 129ms 138ms -9ms -6.5% >250ms and >50% ✅
Refresh round-trip P50 83ms 88ms -5ms -5.7% >250ms and >50% ✅
Refresh round-trip P95 116ms 107ms +9ms +8.4% >300ms and >100% 🔺
Request-to-first environment P50 36ms 41ms -5ms -12.2% >50ms and >50% ✅
Request-to-first environment P95 72ms 49ms +23ms +46.9% >100ms and >100% 🔺
Cold refresh round-trip P50 129ms 139ms -10ms -7.2% >600ms and >50% ✅
Workload PR Baseline
Environments 10 10
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

Comment thread .github/workflows/perf-baseline.yml Resolved
Comment thread .github/workflows/perf-tests.yml Resolved
@rchiodo

Rich Chiodo (rchiodo) commented Nov 16, 2026 •

Copy link
Copy Markdown

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Copy link
Copy Markdown

Reviewed commit: 31901b8be5484c17820bcd8f58abb941cad2e807

Result: 🔴 could-not-verify

Verification details

Verification: Automated verification is best effort. The PR author remains responsible for testing.

Summary: [unavailable] Container verification could not start and local execution was not authorized for this PR HEAD: C:\Program Files\RedHat\Podman\podman.EXE --connection pyrx-automation failed: stderr: Error: unable to start container "575ea93592e3681d3f53b5e051b148144e28dc7e37d0d43735a2afae37fc71f6": crun: open `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

Test runs: none recorded.

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Nov 16, 2026
Move shared measurement capture and provenance checks into an argument-driven script while retaining workload-specific validation. Stage the tested helper revision outside the discovery tree before checking out old coverage source. Retain Intel performance and add a native ARM64 pair to PR and baseline matrices without changing quality budgets.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (macOS ARM64)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 45ms 45ms +0ms +0.0% >100ms and >50% ➖
Server startup P95 65ms 73ms -8ms -11.0% >750ms and >100% ✅
Discovery duration P50 81ms 87ms -6ms -6.9% >100ms and >50% ✅
Discovery duration P95 114ms 105ms +9ms +8.6% >300ms and >100% 🔺
Startup-to-first environment P50 67ms 70ms -3ms -4.3% >150ms and >50% ✅
Startup-to-first environment P95 91ms 83ms +8ms +9.6% >250ms and >100% 🔺
Cold discovery duration P50 118ms 119ms -1ms -0.8% >250ms and >50% ✅
Refresh round-trip P50 82ms 88ms -6ms -6.8% >250ms and >50% ✅
Refresh round-trip P95 114ms 105ms +9ms +8.6% >300ms and >100% 🔺
Request-to-first environment P50 25ms 28ms -3ms -10.7% >50ms and >50% ✅
Request-to-first environment P95 48ms 38ms +10ms +26.3% >100ms and >100% 🔺
Cold refresh round-trip P50 119ms 119ms +0ms +0.0% >600ms and >50% ➖
Workload PR Baseline
Environments 12 12
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

Comment thread scripts/quality_measurement.py Resolved
@StellaHuang95

Copy link
Copy Markdown
Contributor Author

Regarding the automated could-not-verify report for 31901b8: the verifier's Podman container failed before tests could start, so that report is not evidence of a repository test failure. No repository workaround for the reviewer's container is included.

The review follow-up is now pushed as locally GPG-signed f10594f. The PR description records the current evidence: local helper tests, both refactored coverage lanes, all four paired performance lanes, and the four-platform publisher pass. It also explicitly records the two remaining CI blockers: native macOS raw coverage-hit variability and the Intel macOS discovery/direct-identification symlink mismatch. I am not claiming the whole PR is green or weakening either gate. All four inline feedback threads have replies and are resolved.

🤖 AI-authored reply via Agency Finish PR.

@edvilme
Eduardo Villalpando Mello (edvilme) merged commit 5ac67ce into main Nov 16, 2026
47 of 50 checks passed
@edvilme
Eduardo Villalpando Mello (edvilme) deleted the stellahuang95-ci-infrastructure-repair branch November 16, 2026 23:23
Stella Huang (StellaHuang95) added a commit that referenced this pull request Nov 17, 2026
## Summary

Independent follow-up to merged #579, based on
`5ac67ce6dc78dd24c2d0e766bd9b07c1f161721a`. No production locator,
production wait/shutdown semantics, coverage comparator, workflow,
tolerance, workload, or performance lane is changed.

- Replace the duplicated ARM64 substring exclusion in the functional
test with one comparison helper. Only `MacPythonOrg` aliases matching
`/Users/runner/hostedtoolcache/Python/<major.minor.patch>/{x64,arm64}/bin/python`
qualify, and each must resolve to that environment's executable before
exclusion on either side. Different/missing targets fail; ordinary
aliases and metadata comparison remain intact. Focused cases cover both
architectures, unrelated/malformed paths, non-ASCII/non-UTF8 components,
and different/broken targets.
- Make cache owner/value/None/panic tests observe an actual pending
waiter before publication. The instance-local observation is
`cfg(test)`-only and bounded; the existing owner guard still wakes
waiters on unwind.
- Make output tests observe the initial empty-queue wait for each writer
instantiation, and signal blocking test writers/serialization while
holding their release mutex. Test-only observations cannot be overtaken
by send/close/release; production remains unchanged.
- Exercise descendant-termination polling while a real file lease is
held until the pending poll is observed. Document these comparison and
synchronization contracts.

## Controlled native evidence

The merged-main Intel failure independently reproduces the extra x64
toolcache alias: [run 43978821541 / job
133689660257](https://github.com/microsoft/python-environment-tools/actions/runs/43978821541/job/133689660257).

Before changing coverage behavior, [run 43981161006, attempt
1](https://github.com/microsoft/python-environment-tools/actions/runs/43981161006)
measured **the same unchanged main commit** in two separate
source/target trees in predeclared A/B/B/A order. Artifact
**14283815268**, `macos-aa-43981161006-1`, retains full LLVM JSON with
function/instantiation regions, LCOV, repeated exports, raw profiles,
contexts, and isolated child proof.

| Measurement | A1 | B1 | B2 | A2 |
| --- | ---: | ---: | ---: | ---: |
| Unchanged main LH / LF | 18146 / 22512 | 18146 / 22512 | 18149 / 22512
| 18146 / 22512 |
| Patched LH / LF | 18202 / 22563 | 18202 / 22563 | 18202 / 22563 |
18201 / 22563 |

The unchanged **same-path B1/B2** comparison fails at -0.013pp.
Re-exporting the same profiles is byte-identical in every pass. LLVM
regions identify cache waiter paths and `writer_loop<FailingWriter>`
scheduling-dependent hits; a positive union of `DA` lines hides the
latter instantiation difference. This establishes runtime path
variability in this experiment, not a compiler/exporter defect or a
revision-dependent regression. Original artifact **14275796507**
separately identifies the descendant lease's three missed pending-poll
lines.

[Patched run 43983235000, attempt
1](https://github.com/microsoft/python-environment-tools/actions/runs/43983235000),
artifact **14284656224**, `macos-aa-43983235000-1`, passes all four
predeclared raw comparisons. Cache and output summaries are identical
across its four passes. Residual spread is one test-helper line
(0.0044pp), retained without normalization and below the **unchanged
0.01pp** gate. Function totals are constant within each experiment: main
1826/2205; patched 1829/2208.

Both experiments use Rust 1.100.0 / LLVM 23.1.5 / cargo-llvm-cov 0.9.1
on native ARM64 macOS. Every isolated subprocess proof remains idle=0 /
info=1 at handler, transport, and writer. Temporary dispatch-only
experiment changes were removed completely from the final diff.

## Review and validation

- Thorough independent custom **Reviewer** review of the final five-file
patch: **LGTM, no actionable high-confidence bugs**; specifically
examined alias scope, lock ordering, bounded observations, unwind
cleanup, platform gating, and coverage integrity.
- Affected Windows library tests: 157 passed; local alias path
regressions: 2 passed. The malformed-ARM64 retention case failed with
the old substring implementation before the fix.
- Required `cargo fmt --all` and `cargo clippy --all -- -D warnings`,
additional changed-package test-target Clippy, and 17 workflow-contract
tests passed.
- Four complete native default-feature workspace measurements of the
patched Rust source passed, including Unix filesystem identity cases and
subprocess proof.

### Hosted PR validation at `cc3379ba77b7e5221b72824f9d8a560d4802424c`

All six repository workflows passed on their first PR attempt, as did
the external policy check:

| Validation | Evidence | Result |
| --- | --- | --- |
| Native Intel and ARM64 installed-manager tests, both native builds,
full functional matrix | [PR/CI Check
43984292297](https://github.com/microsoft/python-environment-tools/actions/runs/43984292297);
Intel job `133708333703`, ARM64 job `133708333963` | Pass; original
Intel `verify_validity_of_discovered_envs ... ok`, its suite 6 passed /
0 failed |
| Native exact-main raw coverage and isolated child proof | [Native
coverage
43984292202](https://github.com/microsoft/python-environment-tools/actions/runs/43984292202),
artifact **14284442352** | Pass; HEAD 18201/22563 lines and 1829/2208
functions vs main 18146/22512 and 1826/2205; **+0.062pp lines / +0.023pp
functions**, isolated witnesses idle=0/info=1 |
| Linux and Windows exact-base coverage/context gates | [Coverage
43984292251](https://github.com/microsoft/python-environment-tools/actions/runs/43984292251)
| Pass |
| All four performance lanes, including native Intel and ARM64 macOS |
[Performance
43984292229](https://github.com/microsoft/python-environment-tools/actions/runs/43984292229)
| Pass |
| Three-platform Clippy and session benchmarks | [Lint
43984292357](https://github.com/microsoft/python-environment-tools/actions/runs/43984292357),
[benchmarks
43984292289](https://github.com/microsoft/python-environment-tools/actions/runs/43984292289)
| Pass |
| External PR/signing policy | [VS Code PR
Check](https://github.com/microsoft/vscode-engineering/actions/runs/43987471692)
| Pass |

All commits are locally GPG-signed and GitHub-verified. No merge or
auto-merge is enabled; no merge is requested or authorized.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants