You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PET's macOS performance baseline has a deterministic cold-refresh tail near 30 seconds that is hidden by healthy P50 values.
Evidence
Across 11 successful main-branch performance baselines from July 24 through August 6, 2026:
macOS full-refresh P50 median: 137 ms
macOS full-refresh P95 median: 30,291 ms
Latest baseline (8bd5fb4, run 31071711968): P50 142 ms, P95 30,291 ms, mean 8,642 ms
The latest raw iterations were:
30,370 ms (9 environments)
30,195 ms (9 environments)
24,927 ms (10 environments)
then 139, 131, 110, 144, 105, 128, and 174 ms (10 environments)
The same cache directory is reused while each iteration starts a fresh PET server. The first three processes are slow and initially miss one environment, after which the disk cache is warm. Environment inventory is otherwise stable (10 environments, 1 manager), so this is not explained by changing runner inventory.
Investigation requirements
Attribute each slow iteration to a refresh phase and locator using existing RefreshProgress/RefreshPerformance data or equivalent privacy-safe instrumentation.
Determine whether the wait is interpreter spawning, macOS locator probing, cache population, subprocess/pipe behavior, or another blocking path.
Verify every file-based detection path before retaining any spawn fallback.
Measure exact spawn/read operation counts in regression tests.
Acceptance criteria
Root cause is demonstrated with phase/locator and operation-count evidence.
A cold PET process discovers the complete expected environment inventory on its first refresh.
macOS cold-refresh tail no longer approaches the 15/30-second timeout boundaries.
A realistic macOS CI P95 threshold is established and enforced by the snapshot gate.
Median performance does not regress on macOS, Windows, or Linux.
Fix includes behavior tests and performance/operation-count tests.
No extra Python or manager process spawning is introduced.
Diagnostic run 31423551324 on head f9485d0 established the root cause with phase, locator, and operation-count evidence:
First three iterations each emitted exactly two interpreter-probe timeouts, both classified as usrBinPython3.
Iterations: ~30.3s, ~30.2s, ~30.3s; each found only 9 environments.
Path P95: 30,308 ms.
MacCommandLineTools P95: 15,185 ms.
Later iterations had no probe timeouts and completed in 182-294ms with 10 environments.
The two /usr/bin/python3 calls share the same interpreter cache entry. MacCommandLineTools::try_from() starts a 15-second spawn to enrich the alias while PATH discovery asks to resolve the same shim. The cache-entry mutex serializes them: PATH waits for the locator probe, then performs its own 15-second probe after the first timeout, producing the ~30-second phase.
The product fix should therefore:
Resolve Apple /usr/bin/python* shims from the active developer directory using filesystem state (DEVELOPER_DIR / /var/db/xcode_select_link / standard fallbacks).
Let MacXCode or MacCommandLineTools identify the selected shim without spawning it.
Skip spawning unresolved Apple system shims; a non-selected or unavailable shim is not a usable Python environment.
Remove the duplicate /usr/bin/python3 alias-enrichment spawn from the macOS locators.
Add operation-count tests proving zero interpreter spawns for these paths, plus cross-platform behavior tests.
Linux and Windows also passed, retained identical environment inventories, and had no interpreter probe timeouts. A second identical run is in progress to distinguish normal macOS median variance from code drift before opening the PR.
Repeat validation run 31426630835 on the same commit also passed every platform gate:
macOS iteration 1 found all 10 environments; every iteration retained the complete inventory.
0 interpreter probe timeouts.
Full refresh P95: 2,056ms.
path P95: 2,055ms.
MacCommandLineTools P95: 130ms.
macOS P50 remained within the enforced platform budget; Linux and Windows inventories/performance gates also passed.
Across both fixed runs, the deterministic 15s/30s tail did not recur. The remaining cold-run variance is ordinary runner startup/build-host noise rather than interpreter probing.
Full CI, lint, CodeQL, Linux/Windows coverage, and enforced performance gates pass.
Exact-head Copilot review covered 7/7 files with 0 comments and 0 unresolved threads.
macOS repeat: P50 120ms vs 99ms baseline; P95 1,256ms vs 30,320ms baseline (95.9% lower); time-to-first P50 84ms vs 99ms.
Windows repeat: P50 138ms vs 124ms; P95 1,224ms vs 2,533ms. Linux P50 53ms vs 55ms.
Every measured iteration retained the complete inventory (macOS/Windows 10, Linux 5); interpreter probe timeout maps are empty on all platforms.
Untimed diagnostics still retain 10 phase/locator samples; macOS MacCommandLineTools P95 is 35ms.
Coverage increased: Linux lines/functions +0.121pp/+0.015pp; Windows +0.142pp/+0.066pp.
Hosted cold-start variance remains visible (notably first-iteration startup), but two final-head-equivalent gated runs and the exact-head repeat show no material median regression. The deterministic 15s/30s interpreter-probe tail does not recur.
Activity
perf: count interpreter probe timeouts (#504)
karthiknadig commentedon Aug 10, 2026
Diagnostic run
31423551324on headf9485d0established the root cause with phase, locator, and operation-count evidence:usrBinPython3.PathP95: 30,308 ms.MacCommandLineToolsP95: 15,185 ms.The two
/usr/bin/python3calls share the same interpreter cache entry.MacCommandLineTools::try_from()starts a 15-second spawn to enrich the alias while PATH discovery asks to resolve the same shim. The cache-entry mutex serializes them: PATH waits for the locator probe, then performs its own 15-second probe after the first timeout, producing the ~30-second phase.The product fix should therefore:
/usr/bin/python*shims from the active developer directory using filesystem state (DEVELOPER_DIR//var/db/xcode_select_link/ standard fallbacks)./usr/bin/python3alias-enrichment spawn from the macOS locators.perf: resolve macOS Python shims without spawning (#504)
karthiknadig commentedon Aug 10, 2026
First post-fix validation run
31425828087onb923035passed all platform gates.macOS results:
PathP95: 1,436ms (previously ~30,308ms).MacCommandLineToolsP95: 137ms (previously ~15,185ms).Linux and Windows also passed, retained identical environment inventories, and had no interpreter probe timeouts. A second identical run is in progress to distinguish normal macOS median variance from code drift before opening the PR.
karthiknadig commentedon Aug 10, 2026
Repeat validation run
31426630835on the same commit also passed every platform gate:pathP95: 2,055ms.MacCommandLineToolsP95: 130ms.Across both fixed runs, the deterministic 15s/30s tail did not recur. The remaining cold-run variance is ordinary runner startup/build-host noise rather than interpreter probing.
chore: sync issue #504 with main
test: cover public macOS resolver guards (#504)
test: isolate performance diagnostics from timing (#504)
test: retain interpreter timeout diagnostics (#504)
karthiknadig commentedon Aug 10, 2026
Final quality conclusion for PR #506 at
c0f026e:MacCommandLineToolsP95 is 35ms.Hosted cold-start variance remains visible (notably first-iteration startup), but two final-head-equivalent gated runs and the exact-head repeat show no material median regression. The deterministic 15s/30s interpreter-probe tail does not recur.
perf: eliminate macOS discovery timeout tail (Fixes microsoft#504) (m…