Skip to content

ci: pin cargo-llvm-cov installer to release SHA - #578

Closed
Stella Huang (StellaHuang95) wants to merge 1 commit into
mainfrom
stellahuang95-repository-sync
Closed

Stella Huang (StellaHuang95) wants to merge 1 commit into
mainfrom
stellahuang95-repository-sync

Conversation

@StellaHuang95

Copy link
Copy Markdown
Contributor

Pin the remaining mutable taiki-e/install-action@cargo-llvm-cov reference in all three coverage workflows (coverage.yml, coverage-baseline.yml, and coverage-macos.yml) to ab49d3c64680143e184118bed66a46419fc15519, the commit for v2.90.5. Add an explicit tool: cargo-llvm-cov input and a human-readable version comment.

Use the versioned release rather than the generated tool-tag commit: the action maintainer discourages pinning tool tags by hash because those generated commits become unreferenced after subsequent releases. The selected release commit is the parent of the current tool-tag commit, and their only difference is the tool input default; specifying that input explicitly preserves the existing installation behavior.

All 60 action references across the repository's nine workflows now use full-length commit SHAs. Existing weekly grouped Dependabot updates and the seven-day cooldown are unchanged.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 0ms 0ms +0ms +0.0% >5ms and >100% ➖
Server startup P95 0ms 1ms -1ms -100.0% >50ms and >200% ✅
Discovery duration P50 38ms 38ms +0ms +0.0% >25ms and >30% ➖
Discovery duration P95 41ms 41ms +0ms +0.0% >50ms and >50% ➖
Startup-to-first environment P50 8ms 7ms +1ms +14.3% >20ms and >100% 🔺
Startup-to-first environment P95 10ms 9ms +1ms +11.1% >25ms and >100% 🔺
Cold discovery duration P50 88ms 86ms +2ms +2.3% >100ms and >50% 🔺
Refresh round-trip P50 38ms 38ms +0ms +0.0% >25ms and >30% ➖
Refresh round-trip P95 41ms 41ms +0ms +0.0% >50ms and >50% ➖
Request-to-first environment P50 7ms 7ms +0ms +0.0% >20ms and >100% ➖
Request-to-first environment P95 9ms 8ms +1ms +12.5% >25ms and >100% 🔺
Cold refresh round-trip P50 88ms 86ms +2ms +2.3% >100ms and >50% 🔺
Workload PR Baseline
Environments 5 5
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 85.506% 85.511% -0.004pp
Functions 88.038% 88.038% +0.000pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 9ms 9ms +0ms +0.0% >10ms and >50% ➖
Server startup P95 12ms 13ms -1ms -7.7% >50ms and >100% ✅
Discovery duration P50 151ms 154ms -3ms -1.9% >150ms and >50% ✅
Discovery duration P95 158ms 165ms -7ms -4.2% >250ms and >100% ✅
Startup-to-first environment P50 22ms 24ms -2ms -8.3% >25ms and >50% ✅
Startup-to-first environment P95 27ms 36ms -9ms -25.0% >100ms and >100% ✅
Cold discovery duration P50 151ms 153ms -2ms -1.3% >150ms and >50% ✅
Refresh round-trip P50 151ms 154ms -3ms -1.9% >150ms and >50% ✅
Refresh round-trip P95 158ms 166ms -8ms -4.8% >250ms and >100% ✅
Request-to-first environment P50 13ms 14ms -1ms -7.1% >25ms and >50% ✅
Request-to-first environment P95 16ms 25ms -9ms -36.0% >100ms and >100% ✅
Cold refresh round-trip P50 151ms 154ms -3ms -1.9% >150ms and >50% ✅
Workload PR Baseline
Environments 8 8
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 83.028% 83.028% +0.000pp
Functions 85.379% 85.379% +0.000pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 60ms 60ms +0ms +0.0% >100ms and >50% ➖
Server startup P95 69ms 303ms -234ms -77.2% >750ms and >100% ✅
Discovery duration P50 99ms 105ms -6ms -5.7% >100ms and >50% ✅
Discovery duration P95 120ms 114ms +6ms +5.3% >300ms and >100% 🔺
Startup-to-first environment P50 107ms 107ms +0ms +0.0% >150ms and >50% ➖
Startup-to-first environment P95 140ms 129ms +11ms +8.5% >250ms and >100% 🔺
Cold discovery duration P50 162ms 151ms +11ms +7.3% >250ms and >50% 🔺
Refresh round-trip P50 100ms 105ms -5ms -4.8% >250ms and >50% ✅
Refresh round-trip P95 121ms 115ms +6ms +5.2% >300ms and >100% 🔺
Request-to-first environment P50 49ms 49ms +0ms +0.0% >50ms and >50% ➖
Request-to-first environment P95 65ms 56ms +9ms +16.1% >100ms and >100% 🔺
Cold refresh round-trip P50 162ms 152ms +10ms +6.6% >600ms and >50% 🔺
Workload PR Baseline
Environments 10 10
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@rchiodo

Rich Chiodo (rchiodo) commented Nov 16, 2026 •

Copy link
Copy Markdown

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Rich Chiodo (rchiodo) commented Nov 16, 2026 •

Copy link
Copy Markdown

Reviewed commit: dd807a3340ba9fb6d2f5aa3dc9627a218fe8c07c

Result: 🔴 could-not-verify

Verification details

Verification: Automated verification is best effort. The PR author remains responsible for testing.

Summary: [unavailable] Container verification could not start and local execution was not authorized for this PR HEAD: Podman CLI is not installed or is not on PATH.

Test runs: none recorded.

Eduardo Villalpando Mello (edvilme) pushed a commit that referenced this pull request Nov 16, 2026
## Summary

- Provision Linux Miniconda explicitly instead of assuming `$CONDA` is
preinstalled, without creating or activating an extra environment.
Repair both PR workflows and baseline publishers.
- Replace retired macOS 14 runners with supported macOS 15 ARM64
runners, and run x86_64 workloads on `macos-15-intel`. Match Conda
installers to native architecture and retain both architectures'
functional/build coverage.
- Measure the exact PR base afresh. Coverage uses separate clean
base/head jobs because Unix Conda CI tests create global environments
and install Mamba into base. Performance builds each revision and
harness in its own worktree, then measures both on the same runner
against the same discovery workspace and fixtures.
- Validate actual source SHAs, run-attempt-qualified coverage artifacts,
matching nonempty measurement contexts, and successful
test/proof/extraction outcomes before invoking the unchanged
comparators.
- Address review feedback by extracting argument-driven
staging/capture/comparison into `scripts/quality_measurement.py` and
adding a distinct native ARM64 lane to both performance workflows while
retaining Intel.

## Root cause and impact

Before this repair, Linux jobs failed before testing because Ubuntu
26.04 no longer supplied the Conda installation assumed by the
workflows. The cancelled macOS jobs never received a runner: their macOS
14 configuration retired on November 2.

The original Windows numerical coverage failure compared Rust
1.99.1/LLVM 23.1.1 with Rust 1.100.0/LLVM 23.1.5 and different runner
images. With the same 96 Rust source files, LCOV denominators changed
from LF 23,480/FNF 2,231 to LF 23,288/FNF 2,254. This demonstrates
measurement-environment drift, not that the compiler alone explains
every changed hit.

The original macOS startup-P95 outlier is not treated as a proven
product defect. Its one unchanged-head rerun passed the timing budgets
but failed inventory matching: 12 environments versus the historical
baseline's 11. Fresh comparisons address that comparability problem
without discarding slow samples or normalizing inventories away.

## Preserved invariants and review follow-up

- No Rust/product, regression-budget, or snapshot-schema changes. The
0.01 percentage-point coverage tolerance, all performance budgets,
required samples, and inventory checks remain intact.
- Native ARM64 macOS coverage retains its default-feature workload and
same-runner exact-base comparison. Installed-manager fixtures,
environments without Python, subprocess proof, and fail-closed reporting
remain required.
- The extracted helper stages only the tested workflow's two immutable
script blobs plus a revision marker outside the discovery tree before
coverage checks out the measured source. An older base need not contain
the new helper. `helper_revision` and the measured source SHA are
validated separately; head Rust source/binaries are never substituted
for the base.
- Coverage and performance share common capture/provenance logic but
retain their different required tools and fixtures. Numerical gates
remain in the unchanged `quality_snapshot.py`.
- Native ARM64 performance has its own artifacts/comment lane and
freshly measured exact base. It uses the existing macOS budgets; Intel
remains included.
- Historical baseline workflows remain publishers for inspection. PR
gates no longer depend on their older artifacts. Fresh comparisons add a
real base measurement per platform.
- Separate from #578: its branch and installer-pinning changes are
untouched. No merge is requested.

## Current validation

Current head: `f10594fd644f112fa609dce548feab29d0dd4c92`, a new
**locally GPG-signed** commit preserving parent `31901b8`. Both local
GnuPG verification and GitHub's OpenPGP verification succeed. The
trusted-team-key signing policy recognizes the signature; `VS Code PR
Check` now passes with the required collaborator approval.

Actual measurements use exact base
`02dd29e49e3c71d4ec06e1072d6331494a12e01d` and tested merge
`d6d49a1ad7080a4e9bcda6259cd60a11fa3ab384` containing the current head.

- Local Python suite: 116 tests, one skipped. The parent independently
ran all 17 quality-workflow tests from the immutable signed revision.
Tests exercise the real CLI and reject wrong revisions, missing staged
helpers/dependencies, mismatched contexts, failed execution/extraction,
and accidental reuse of head as base.
- actionlint passes all three follow-up workflows; the original
seven-workflow repair was also linted.
- [Refactored Linux/Windows
coverage](https://github.com/microsoft/python-environment-tools/actions/runs/43964532818)
passes. Downloaded artifacts verify the old-base/new-helper case,
matching contexts, and required zero-idle/positive-info
handler/transport/writer witnesses for both revisions. Line deltas are
Linux +0.004445pp and Windows -0.004294pp; function deltas are zero,
within unchanged budgets.
- [All four paired performance
lanes](https://github.com/microsoft/python-environment-tools/actions/runs/43964532800)
pass all 12 gates with all nine distributions containing ten samples
each. Base/head environment inventories match: Linux 5/5, Windows 8/8,
Intel macOS 10/10, ARM64 macOS 12/12; managers are 1/1 throughout. ARM64
startup P95 is 73ms base versus 65ms head.
- [The updated performance
publisher](https://github.com/microsoft/python-environment-tools/actions/runs/43964849683)
passes on all four platforms. Its ARM64 VM independently has 11
environments/1 manager; that legitimate cross-run difference is
retained, not compared against the paired workload or normalized away.

## Remaining CI blockers

The review changes are pushed and all four reviewer threads have replies
and are resolved, but **the PR is not fully green**:

- [Native ARM64
coverage](https://github.com/microsoft/python-environment-tools/actions/runs/43964532819)
passes functional tests, isolated proof, and reporting but fails the raw
line gate by -0.031pp. Exactly one unchanged diagnostic rerun also
failed: head LH/LF 18,146/22,512 versus base 18,153/22,512, with
functions identical at 1,826/2,205. The seven-hit discrepancy moved
between unchanged source paths across attempts; the second attempt's
differing output.rs summaries have identical positive DA line sets. Both
failed artifacts are retained. This is unresolved runtime/summary-hit
variability, not a missing helper or removed test. No further rerun or
tolerance change was made.
- [Intel macOS functional
tests](https://github.com/microsoft/python-environment-tools/actions/runs/43964532730/job/133640357939)
fail `verify_validity_of_discovered_envs` at `ci_test.rs:629`: full
discovery includes one additional hosted-toolcache x64 Python symlink
that direct `try_from` does not return. Interpreter
identity/version/prefix/manager/architecture agree. Existing CI
normalization excludes ARM64 toolcache aliases but not x64. The initial
repair passed on the same image/Python version; why the extra alias
appears in this run remains unproven. This follow-up changes neither
`pr-check.yml` nor Rust and that job does not invoke the helper. No
product-locator workaround or blanket retry was made.

These remaining reliability/alias-causality issues are reported
explicitly rather than weakening gates or silently expanding the
maintenance-feedback fix.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Performance Report (macOS ARM64)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 39ms 46ms -7ms -15.2% >100ms and >50% ✅
Server startup P95 56ms 73ms -17ms -23.3% >750ms and >100% ✅
Discovery duration P50 72ms 100ms -28ms -28.0% >100ms and >50% ✅
Discovery duration P95 94ms 117ms -23ms -19.7% >300ms and >100% ✅
Startup-to-first environment P50 62ms 74ms -12ms -16.2% >150ms and >50% ✅
Startup-to-first environment P95 74ms 113ms -39ms -34.5% >250ms and >100% ✅
Cold discovery duration P50 97ms 124ms -27ms -21.8% >250ms and >50% ✅
Refresh round-trip P50 73ms 101ms -28ms -27.7% >250ms and >50% ✅
Refresh round-trip P95 94ms 117ms -23ms -19.7% >300ms and >100% ✅
Request-to-first environment P50 25ms 27ms -2ms -7.4% >50ms and >50% ✅
Request-to-first environment P95 31ms 47ms -16ms -34.0% >100ms and >100% ✅
Cold refresh round-trip P50 98ms 124ms -26ms -21.0% >600ms and >50% ✅
Workload PR Baseline
Environments 11 11
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Nov 17, 2026
Use the v2.90.5 release commit with an explicit cargo-llvm-cov tool input in all three coverage workflows.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants