Skip to content

test: stabilize native macOS alias and coverage checks - #581

Merged
Stella Huang (StellaHuang95) merged 3 commits into
mainfrom
stellahuang95-macos-ci-reliability
Nov 17, 2026
Merged

Stella Huang (StellaHuang95) merged 3 commits into
mainfrom
stellahuang95-macos-ci-reliability

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Independent follow-up to merged #579, based on 5ac67ce6dc78dd24c2d0e766bd9b07c1f161721a. No production locator, production wait/shutdown semantics, coverage comparator, workflow, tolerance, workload, or performance lane is changed.

  • Replace the duplicated ARM64 substring exclusion in the functional test with one comparison helper. Only MacPythonOrg aliases matching /Users/runner/hostedtoolcache/Python/<major.minor.patch>/{x64,arm64}/bin/python qualify, and each must resolve to that environment's executable before exclusion on either side. Different/missing targets fail; ordinary aliases and metadata comparison remain intact. Focused cases cover both architectures, unrelated/malformed paths, non-ASCII/non-UTF8 components, and different/broken targets.
  • Make cache owner/value/None/panic tests observe an actual pending waiter before publication. The instance-local observation is cfg(test)-only and bounded; the existing owner guard still wakes waiters on unwind.
  • Make output tests observe the initial empty-queue wait for each writer instantiation, and signal blocking test writers/serialization while holding their release mutex. Test-only observations cannot be overtaken by send/close/release; production remains unchanged.
  • Exercise descendant-termination polling while a real file lease is held until the pending poll is observed. Document these comparison and synchronization contracts.

Controlled native evidence

The merged-main Intel failure independently reproduces the extra x64 toolcache alias: run 43978821541 / job 133689660257.

Before changing coverage behavior, run 43981161006, attempt 1 measured the same unchanged main commit in two separate source/target trees in predeclared A/B/B/A order. Artifact 14283815268, macos-aa-43981161006-1, retains full LLVM JSON with function/instantiation regions, LCOV, repeated exports, raw profiles, contexts, and isolated child proof.

Measurement A1 B1 B2 A2
Unchanged main LH / LF 18146 / 22512 18146 / 22512 18149 / 22512 18146 / 22512
Patched LH / LF 18202 / 22563 18202 / 22563 18202 / 22563 18201 / 22563

The unchanged same-path B1/B2 comparison fails at -0.013pp. Re-exporting the same profiles is byte-identical in every pass. LLVM regions identify cache waiter paths and writer_loop<FailingWriter> scheduling-dependent hits; a positive union of DA lines hides the latter instantiation difference. This establishes runtime path variability in this experiment, not a compiler/exporter defect or a revision-dependent regression. Original artifact 14275796507 separately identifies the descendant lease's three missed pending-poll lines.

Patched run 43983235000, attempt 1, artifact 14284656224, macos-aa-43983235000-1, passes all four predeclared raw comparisons. Cache and output summaries are identical across its four passes. Residual spread is one test-helper line (0.0044pp), retained without normalization and below the unchanged 0.01pp gate. Function totals are constant within each experiment: main 1826/2205; patched 1829/2208.

Both experiments use Rust 1.100.0 / LLVM 23.1.5 / cargo-llvm-cov 0.9.1 on native ARM64 macOS. Every isolated subprocess proof remains idle=0 / info=1 at handler, transport, and writer. Temporary dispatch-only experiment changes were removed completely from the final diff.

Review and validation

  • Thorough independent custom Reviewer review of the final five-file patch: LGTM, no actionable high-confidence bugs; specifically examined alias scope, lock ordering, bounded observations, unwind cleanup, platform gating, and coverage integrity.
  • Affected Windows library tests: 157 passed; local alias path regressions: 2 passed. The malformed-ARM64 retention case failed with the old substring implementation before the fix.
  • Required cargo fmt --all and cargo clippy --all -- -D warnings, additional changed-package test-target Clippy, and 17 workflow-contract tests passed.
  • Four complete native default-feature workspace measurements of the patched Rust source passed, including Unix filesystem identity cases and subprocess proof.

Hosted PR validation at cc3379ba77b7e5221b72824f9d8a560d4802424c

All six repository workflows passed on their first PR attempt, as did the external policy check:

Validation Evidence Result
Native Intel and ARM64 installed-manager tests, both native builds, full functional matrix PR/CI Check 43984292297; Intel job 133708333703, ARM64 job 133708333963 Pass; original Intel verify_validity_of_discovered_envs ... ok, its suite 6 passed / 0 failed
Native exact-main raw coverage and isolated child proof Native coverage 43984292202, artifact 14284442352 Pass; HEAD 18201/22563 lines and 1829/2208 functions vs main 18146/22512 and 1826/2205; +0.062pp lines / +0.023pp functions, isolated witnesses idle=0/info=1
Linux and Windows exact-base coverage/context gates Coverage 43984292251 Pass
All four performance lanes, including native Intel and ARM64 macOS Performance 43984292229 Pass
Three-platform Clippy and session benchmarks Lint 43984292357, benchmarks 43984292289 Pass
External PR/signing policy VS Code PR Check Pass

All commits are locally GPG-signed and GitHub-verified. No merge or auto-merge is enabled; no merge is requested or authorized.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Compare only verified setup-python shortcuts across native architectures. Observe cache, output, and descendant lease waits before releasing their owners so LLVM instantiation coverage does not depend on scheduling.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the controlled runs as immutable evidence without adding a permanent diagnostic workflow or changing the native coverage gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 1ms 0ms +1ms +inf% >5ms and >100% 🔺
Server startup P95 1ms 1ms +0ms +0.0% >50ms and >200% ➖
Discovery duration P50 51ms 47ms +4ms +8.5% >25ms and >30% 🔺
Discovery duration P95 53ms 52ms +1ms +1.9% >50ms and >50% 🔺
Startup-to-first environment P50 9ms 8ms +1ms +12.5% >20ms and >100% 🔺
Startup-to-first environment P95 12ms 9ms +3ms +33.3% >25ms and >100% 🔺
Cold discovery duration P50 104ms 99ms +5ms +5.1% >100ms and >50% 🔺
Refresh round-trip P50 51ms 48ms +3ms +6.2% >25ms and >30% 🔺
Refresh round-trip P95 53ms 53ms +0ms +0.0% >50ms and >50% ➖
Request-to-first environment P50 8ms 7ms +1ms +14.3% >20ms and >100% 🔺
Request-to-first environment P95 11ms 8ms +3ms +37.5% >25ms and >100% 🔺
Cold refresh round-trip P50 104ms 99ms +5ms +5.1% >100ms and >50% 🔺
Workload PR Baseline
Environments 5 5
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 85.502% 85.447% +0.055pp
Functions 88.038% 88.022% +0.016pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Performance Report (macOS ARM64)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 26ms 32ms -6ms -18.8% >100ms and >50% ✅
Server startup P95 56ms 51ms +5ms +9.8% >750ms and >100% 🔺
Discovery duration P50 67ms 52ms +15ms +28.8% >100ms and >50% 🔺
Discovery duration P95 85ms 101ms -16ms -15.8% >300ms and >100% ✅
Startup-to-first environment P50 39ms 55ms -16ms -29.1% >150ms and >50% ✅
Startup-to-first environment P95 65ms 75ms -10ms -13.3% >250ms and >100% ✅
Cold discovery duration P50 81ms 87ms -6ms -6.9% >250ms and >50% ✅
Refresh round-trip P50 67ms 53ms +14ms +26.4% >250ms and >50% 🔺
Refresh round-trip P95 85ms 101ms -16ms -15.8% >300ms and >100% ✅
Request-to-first environment P50 15ms 23ms -8ms -34.8% >50ms and >50% ✅
Request-to-first environment P95 32ms 36ms -4ms -11.1% >100ms and >100% ✅
Cold refresh round-trip P50 81ms 87ms -6ms -6.9% >600ms and >50% ✅
Workload PR Baseline
Environments 12 12
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 83.028% 82.961% +0.067pp
Functions 85.379% 85.359% +0.019pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 7ms 7ms +0ms +0.0% >10ms and >50% ➖
Server startup P95 9ms 9ms +0ms +0.0% >50ms and >100% ➖
Discovery duration P50 94ms 92ms +2ms +2.2% >150ms and >50% 🔺
Discovery duration P95 98ms 97ms +1ms +1.0% >250ms and >100% 🔺
Startup-to-first environment P50 15ms 14ms +1ms +7.1% >25ms and >50% 🔺
Startup-to-first environment P95 17ms 18ms -1ms -5.6% >100ms and >100% ✅
Cold discovery duration P50 92ms 93ms -1ms -1.1% >150ms and >50% ✅
Refresh round-trip P50 94ms 93ms +1ms +1.1% >150ms and >50% 🔺
Refresh round-trip P95 99ms 97ms +2ms +2.1% >250ms and >100% 🔺
Request-to-first environment P50 8ms 7ms +1ms +14.3% >25ms and >50% 🔺
Request-to-first environment P95 9ms 11ms -2ms -18.2% >100ms and >100% ✅
Cold refresh round-trip P50 92ms 94ms -2ms -2.1% >150ms and >50% ✅
Workload PR Baseline
Environments 8 8
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 68ms 76ms -8ms -10.5% >100ms and >50% ✅
Server startup P95 84ms 96ms -12ms -12.5% >750ms and >100% ✅
Discovery duration P50 121ms 129ms -8ms -6.2% >100ms and >50% ✅
Discovery duration P95 158ms 156ms +2ms +1.3% >300ms and >100% 🔺
Startup-to-first environment P50 119ms 120ms -1ms -0.8% >150ms and >50% ✅
Startup-to-first environment P95 168ms 140ms +28ms +20.0% >250ms and >100% 🔺
Cold discovery duration P50 187ms 195ms -8ms -4.1% >250ms and >50% ✅
Refresh round-trip P50 121ms 130ms -9ms -6.9% >250ms and >50% ✅
Refresh round-trip P95 158ms 157ms +1ms +0.6% >300ms and >100% 🔺
Request-to-first environment P50 50ms 53ms -3ms -5.7% >50ms and >50% ✅
Request-to-first environment P95 89ms 67ms +22ms +32.8% >100ms and >100% 🔺
Cold refresh round-trip P50 188ms 196ms -8ms -4.1% >600ms and >50% ✅
Workload PR Baseline
Environments 10 10
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@rchiodo

Copy link
Copy Markdown

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Copy link
Copy Markdown

Reviewed commit: cc3379ba77b7e5221b72824f9d8a560d4802424c

Result: 🔴 could-not-verify

Verification details

Verification: Automated verification is best effort. The PR author remains responsible for testing.

Summary: [unavailable] Container verification could not start and local execution was not authorized for this PR HEAD: Podman CLI is not installed or is not on PATH.

Test runs: none recorded.

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Nov 17, 2026
@StellaHuang95
Stella Huang (StellaHuang95) merged commit 25ff675 into main Nov 17, 2026
45 checks passed
@StellaHuang95
Stella Huang (StellaHuang95) deleted the stellahuang95-macos-ci-reliability branch November 17, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants