Conversation
Moon migration belongs in PR NVIDIA#2659. Restore the pre-Moon selective-CI planner and workflows from 727ef59.
rwgk
added
enhancement
|
/ok to test 17e19e2 |
Work Report — GPT-6-Astra ultra — Autonomous drive to merge readinessOutcomeThe engineering work is complete, and I am comfortable with merging this implementation after the required renewed reviewer approval. PR #2737 is marked ready for review at The chosen direction is implemented: maintain the two supported CUDA ABI-major source roots together on Expand for detailsDate: September 29, 2026. Requested report start label: 0640 America/Los_Angeles. Starting point and scope
Added product commits
The user explicitly approved retaining the existing fork head as an exception to the standing upstream-branch convention, so all product commits were pushed to Decisions resolvedBranches and maintenance scope
Major-line maintenance can include explicitly selected API, Python, or platform support. Once a patch release is frozen, only approved fixes and release prerequisites enter that release. The checklist records this scope rather than treating the word "maintenance" as a complete release policy. The manual backport workflow validates a merged source PR and one existing non-default target branch. The pinned action creates a reviewable PR. Label-derived targets are disabled; there is no automatic merge. Workflow-file changes need appropriate manual credentials, and generated changes need target-specific regeneration. Branch-first emergency repairs require a linked return to Durable procedure: RELEASE-bindings.md. Layout, registry, and package boundariesKeep the explicit CUDA 12 physical root and separate lifecycle roles. The supported shape is exactly one current and one maintenance major. Retaining the mapping avoids scattering source paths through workflows; it does not implement arbitrary numbers of supported public lines. A future 13/14 rollover must preserve the CUDA 13 source in a separate root before introducing CUDA 14. Keep explicit minor-family SCM selectors so source builds do not accidentally select a prior minor's tags. Keep the standard setuptools-scm parser. The standalone metapackage's small selector/dependency policy is intentional because its sdist cannot import sibling source trees or repository CI metadata. The new guard checks the duplicated selectors and maintenance fallback against the actual package metadata. Release independence and historical compatibilityNormal development CI includes affected dependents and can build both Core ABI variants. A bindings release tag selects one bindings root and its matching metapackage, uses published Pathfinder, and does not require another-major or Core release. Stabilization branches isolate unfinished development by freezing a tested source baseline; they do not waive required checks. The tagged registry is authoritative when present. Historical trees without it retain a bounded compatibility path, with separate source and workflow-control revisions. Expired artifacts and indefinitely retired configurations are not promised to remain releasable. Cross-root review and cybindHandwritten changes require explicit applicability review in both roots. Semantic equivalence cannot generally be established by comparing bytes. Generated imports should record the generator revision, toolkit inputs, command, and any manual adjustments. Seals certify bytes, not generator provenance. Continuous toolkit QA and broader cybind asset/consumer improvements remain independent follow-ups. This change does not need a generator overhaul or QA directory move to establish the new source ownership. The final generated payload remains the reviewed v12.9.9 import plus previously disclosed overlays and seal normalization. Concrete defects fixed during this drive
Removed five redundant snapshot/static-agreement tests. Retained negative artifact selection, matrix completeness, exact-tag identity, historical-source, and mixed-workplan tests because ordinary happy-path CI does not exercise those failures. Local validation
Local logs:
Hosted validationThe product head advanced to
After marking the PR ready, a stable, fully paginated check read confirms 121 successful checks, three intentional skips, and none pending or failing. These 124 checks include auxiliary rehearsal checks; normal CI itself has 108 successful jobs. The separate The final-head CI metadata/helper gate independently confirms the same 191 tests and 55 subtests passed. The original complete attempt and the failed-job retry are preserved separately; the following section records the two intermittent Windows failures and the investigation rather than hiding them behind the final green result. Final-head Windows failure investigationThe first observed normal-CI failure was Windows Python 3.14t / CUDA 13.0.2 wheels / A100 MCDM. The Compared the exact same row against main baseline 36e4d40 and prior PR head 59ff7bd: both had the example pass and reported 3,903 passed, 486 skipped, and two expected failures. All three used published bindings 13.0.3, CuPy 14.2.0, NVRTC 13.0.88, runtime 13.0.96, and nvJitLink 13.4.92. Core source, Pathfinder source, the example, and its test have identical Git object identities between the main baseline and final head. The CUDA 12 imported bindings are not installed in that row. This evidence supports a focused diagnostic rerun; it does not identify the native crash's root cause or justify a speculative code workaround. The original attempt completed with 108 jobs: 105 successful, two failed Windows test rows, and their derived The failed-job retry completed successfully at 16:07:31 UTC. GitHub reports all 108 jobs successful: 105 successful first-attempt jobs retain their original execution timestamps, while only the two failed Windows rows and the aggregate gate execute again. The CUDA 13.0 retry explicitly passes the fractal test and reports 3,903 Core tests passed, 486 skipped, two expected failures, and 15 warnings in 276.51 seconds. The CUDA 13.4 retry explicitly passes the VMM growth case and reports 4,060 Core tests passed, 409 skipped, four expected failures, and 112 warnings in 379.06 seconds. The product source is unchanged throughout; no test suppression, relaxed assertion, or reduced matrix was used. The original failure causes remain unresolved, with the separately confirmed existing ownership defect described below. A second job failed in Windows Python 3.14t / CUDA 13.4.2 wheels / A100 MCDM: Queued x86 L4 rows delayed completion and GitHub rejected a job rerun while the containing workflow remained active. Read-only runner status was unavailable (repository runner API permission denied). Historical successful baseline jobs had already shown 78-162 minute L4 queue delays, so the observed queue alone did not establish an outage. The test matrix was preserved. A separate Windows diagnostic run uses controller The diagnostic completed green on its first attempt: all three jobs passed. The CUDA 13.0 row explicitly reports the fractal example Separate pre-existing Core ownership defectThe failure investigation uncovered and independently confirmed a duplicate deallocation attempt in the existing VMM slow-growth path. Buffer ownership captures the resource and size in a C++ deleter. Slow growth explicitly unmaps and frees the old address, then Buffer._clear() resets the owner, invoking its deallocator again. The deallocator can stop at a failed unmap; this is a duplicate cleanup attempt, not proof of a second successful free. Matching destructor warnings appear in both successful baseline runs as well as the failed run. All implicated source files are byte-identical to the main baseline. The failing temporary-reservation address differs from the warning addresses, so the logs do not establish an address-reuse interleaving or prove this defect caused that failure. No common cause with the JIT subprocess crash is established. A separate Core correction should preserve the old mapping and owner while transactionally constructing the new mapping, transfer new-resource ownership exactly once, and cover rollback and retained-view behavior. Simply resetting an ownership flag or explicitly closing the old Buffer can invalidate DLPack exports that retain it. This ownership/API follow-up is outside the branch migration's source changes and is recorded for team follow-up. Exact-source CUDA 12 rehearsalSource candidate: The controller changes only five workflow files in a separate worktree. It pins source checkouts and artifact identities to the product candidate, creates synthetic The rehearsal deliberately verifies rejection of missing synthetic 12.9.99 notes and independently validates real 12.9.9 notes. It does not manufacture a product release note. The intended payload checks cover 18 release wheels, one matching metapackage wheel, wheel METADATA and the stable metapackage's compatible-release bindings requirements (base and All 48 original prerequisites passed: 24 native wheel builds, two sdists, 17 GPU test rows, documentation, three matrix jobs, and source resolution. The original rehearsal's final custom metadata assertion was too strict: it expected one The first validation-only attempt, 36583210291, exposed another harness-only issue: hosted Evidence artifact: Downloaded and independently inspected the evidence: all 19 wheel hashes match, all wheel metadata versions are 12.9.99, and the binary matrix is exactly CPython 3.10, 3.11, 3.12, 3.13, 3.14, and free-threaded 3.14 across Linux x86-64, Linux AArch64, and Windows x86-64. Experimental 3.15/3.15t builds are excluded from publication inputs by the production validator. The source archive hash is This source rehearsal uses a branch push, so it does not establish tag-event run discovery or publication credentials. The separate production dry runs exercise exact-tag lookup with real retained artifacts. Production release dry runsControl revision:
All four completed successfully, including exact-tag CI discovery, notes validation, docs, wheel-matrix checks, and release-artifact preparation. PyPI/TestPyPI publication was skipped; no GitHub release or docs deployment was created. Exact source/artifact lineage: v13.4.3 = Final-source documentation refreshFinal source: All three jobs passed. Sphinx warnings decreased from 96 to 93; the three authored-doc defects are fixed. Remaining generated-API warning categories match the historical v12.9.9 source. Duplicate object descriptions are more numerous because this tree retains more release-note documents. The full docs build is successful, not warning-free. Review dispositionResolved all 45 threads that were open at the initial audit after checking the implementation, documented decisions, relevant validation, and fresh discussion state. GitHub now shows 52 of 52 threads resolved, including the seven that were already resolved. The PR body contains a grouped disposition map with permanent source links; the appendix below records all 45 individual decisions. New comments and thread state were checked before mutations. No thread reply or unrelated issue/comment was posted; the only new PR comments were the two expressly requested Author-side thread resolution is distinct from human approval. No review was dismissed and no approval was represented as granted. The PR was not merged. Remaining boundaries
Final repository and PR state
Appendix: individual review dispositionsThe PR description contains the grouped review map. These individual decisions cover the 45 threads that were open at the initial audit; they do not imply reviewer approval.
|
@brandon-b-miller, GPT-6-Astra ultra implemented a practical compromise. In its own words: Your concern is real: an ordinary CUDA 12 maintenance PR can still encounter CUDA 13/Core CI failures. The proposal retains that integration testing and adds a release path that can proceed from a tested baseline. The compromise has three parts:
This addresses the release-stage coupling and provides a practical way around unrelated breakage on advancing So the intended guarantee is that a bindings release does not require a Core build, test run, or release. It is not a guarantee that every possible Core problem is irrelevant to the process. |
|
@leofang Following your offline question about whether we should continue #2737, I asked Codex for a deep analysis of the Its conclusion: keep #2737’s direction, combined with explicit release branches for stabilization and urgent fixes. The main reasons:
The proposed division is therefore: develop the two supported majors together on The revised PR implements that direction with focused bindings-release CI, a manual backport workflow, and a short release guide. Normal PRs retain Core integration checks, and generated changes still need explicit applicability review. That gives us a concrete reason to finish #2737 while continuing to improve branch automation and generator coordination. |
Integrate main through ab9e20f. Preserve the registry-driven dual-root build and release model while adopting Core bindings floors, uv builds, wheel-source provenance, compiler/cache fixes, and docs deployment concurrency. Port applicable shared build helpers to CUDA 12 and cover schema-2 floors and merged-wheel bootstrap files with regression tests. The only new 12.9.x commit (1635cf2) updates Actions pins; its applicable updates are included without restoring retired branch workflows.
Preserve the configuration around size_t, bool, and string roundtrip tests so random ordering cannot disable compatibility mode or leave invalid settings for later driver opens. Restore values in finally blocks and keep valid defaults for empty string settings rejected by cuFile setters.
|
/ok to test 07845c5 |
@rwgk, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 07845c5 |
|
/ok to test 83ae256 |
|
/ok to test 5b41dd1 |
|
/ok to test be28657 |
|
Independent "full review" with Fable 5 ( Review complete — all five parallel review passes are in, and I verified the two highest-value findings directly in the code. Bottom line: no critical defects; the PR's core machinery (tag-family SCM selection, the self-disabling CUDA 12 fallback, release provenance validation, the registry/metadata consistency checks) holds up under both code reading and actual execution. One major finding in the release workflows should be fixed before merge; the rest are minor robustness/asymmetry items. Findings
SYNCED-block verification (per repo policy) Groups compared: Import fidelity and the rest The cuda_bindings_12 tree is a high-fidelity import of upstream v12.9.9: every generated binding module and generated doc is byte-identical to the tag, and all 79 differing files decompose cleanly into the declared overlays (coredump buffer-lifetime fix, str/bytes nested-resource fix with paired tests, NVML test port, cuFile isolation work, the runtime.pyx.in seal correction — seal checker passes, and monorepo packaging/docs adaptations). No unexplained source changes. Test authorship markers are present on every newly added test checked (directly or via class-level markers). The two stale NVRTC doc entries and the 12.9.7 README links match the known boundaries stated in the PR description. All 52 PR review threads are resolved. The SCM design's key question — "does the custom fallback reliably get out of the way once standard Git-derived versioning can work?" — checks out: the fallback probe uses Open questions
Overall this is unusually carefully engineered for a change of this size — fail-closed defaults, env-var routing of untrusted input, provenance enforced end-to-end from tag to SHA to run to wheel matrix. The recurring structural theme behind most minor findings is invariants enforced by convention in two or three places instead of derived from one source (findings 2, 5, 9). I'd fix finding 1 ( |
|
codex ( CI / Test win-64 / Python 3.15, CUDA 13.4.2 (wheels), GPU l4 (MCDM) Failing This is a test-dependency installation failure. It is connected to PR 2737: the PR adds The job log shows:
PyYAML 6.0.3 has no Python 3.15 wheels, which explains the source build. I recommend using PyYAML’s pure-Python implementation for Windows/Python 3.15. Append this line to the existing Windows Python 3.15 setup step: echo "PYYAML_FORCE_LIBYAML=0" >> "$GITHUB_ENV"PyYAML supports this setting. It preserves the dependency and registry tests while avoiding extension compilation. I verified that the setting builds a pure-Python wheel with an unavailable compiler and that YAML parsing works. That verification used Linux/Python 3.12; the Windows/Python 3.15 job still needs a rerun after the adjustment. The current failure provides no evidence of a CUDA implementation regression. |
Disable the optional LibYAML extension before test dependency installation. Python 3.15 has no matching PyYAML wheel, and GPU test runners do not configure MSVC. Validated a source-built pure-Python wheel with an unavailable compiler; 268 CI-tool and Core tests plus 55 subtests passed against it.
Prevent same-named branches from shadowing release tags when resolving configuration, notes, docs, and source archives. Preserve branch and default-ref behavior for ordinary docs builds. Validated with the pinned checkout implementation and real same-name branch/tag collisions; six cases and actionlint passed.
Require the maintenance selector to exclude the pre-maintenance .0 baseline, even when bindings and metapackage selectors are changed together. Require the current selector to include .0 and reject a current-root fallback_version. Three regression cases failed before the fix. Integrated CI-tool and Core floor tests passed with pure-Python PyYAML.
|
/ok to test 6c3de41 |
|
Fable 5 ( Re-review complete: all three new commits are correct, both actionable findings are properly fixed, and the Python 3.15 Windows job now passes. No new findings. What I verified on the fresh head (6c3de41)
Status of the remaining minor findings — unchanged by these commits, consistent with your note that only the two actionable ones were addressed. The ones I'd still keep on a follow-up list, roughly in order of value: the cuda_bindings_12 pixi I cleaned up the temporary worktree I used for testing; your checkout is untouched (still at |
rwgk
added
the
PR review get-together
|
My concern with this PR is no one is going to do a decent job of reviewing a 302 file change. I'd would highly encourage breaking this change up into smaller changes that can be atomically reviewed and submitted. Reading through the 'Decision Implemented' section seems like those would be good candidates to explore as atomic changes. I'd track them as sub-work of a larger EPIC. cc: @leofang |
rparolin
left a comment
rparolin
left a comment
There was a problem hiding this comment.
Please break this PR up into smaller reviewable chunks.
There is already a narrated "Refreshed six-layer review aid" in the PR description that serves that purpose. |
This branch has not been deployed
7 participants
Summary
Closes #1199. This continues Keith Kraus's original PR #2675, preserving his CUDA 12 source import and initial build, test, and release integration. The replacement became necessary when that PR's temporary base branch was deleted after #2467 merged.
Maintain both supported CUDA ABI-major source trees on
main, with short release branches available for stabilization and urgent fixes. Release each bindings line independently, together with its matchingcuda-pythonmetapackage.cuda_bindings_12/cuda_bindings/The roots build the same
cuda-bindingsdistribution andcuda.bindingsnamespace in separate environments. The CUDA 12 source import includes v12.9.9. The October 3 refresh mergesmainthroughab9e20fd6a525d526d85d529571b3940824e8467and incorporates the subsequent12.9.xworkflow dependency updates through1635cf2c5552bc329497779b73d3ec72075e2434. After merge,12.9.xis retained as historical release evidence.Decisions Implemented
cuda_bindings_12/for CUDA 12 andcuda_bindings/for CUDA 13.ci/versions.ymlrecords their package paths, toolkit versions, and current/maintenance roles. Support exactly one current and one maintenance root with distinct CUDA majors. Naming the roots for a future major-version rollover remains a future decision.main; cut short stabilization branches from a tested commit when needed. Urgent fixes can start on the frozen branch, with a linked follow-up PR to reconcile the fix back intomain. A new manual backport workflow accepts one merged PR and one existing non-default target branch, opens a reviewable PR, and leaves CI approval and merging to the normal process. The historical12.9.xbranch is not resumed.mainis unfinished; it does not waive required validation. — For details, see this comment below.setuptools-scm, with each bindings line selecting its own tag family. A temporary fallback handles CUDA 12 builds before an eligible tag is reachable. The standalone metapackage carries matching build metadata, and pre-commit/CI checks prevent those copies from drifting. See the SCM and packaging review guide below.ci/versions.ymlat the release tag. The new release tooling supports tags containing this configuration; older tags require compatible historical tooling. For normal releases, CI: Release requires each component's exact-version, nonempty notes in the tagged source, including separate metapackage notes. The existing exemption for.postNreleases remains.SYNCEDblocks identify deliberately identical snippets; agent instructions require checking every copy during development and review. The maintenance policy requires future generated imports to record the cybind revision, toolkit inputs, commands, and manual adjustments. File seals verify file contents but do not establish generation provenance. Continuous toolkit QA, broader cybind changes, and a uniform pre-commit check forSYNCEDblocks remain separate work.SCM and packaging review guide
Click to un/expand
setuptools-scmhandles parsing and version progression. CUDA 12 excludes the oldv12.9.0baseline.The bootstrap regression cases cover no eligible tag, the excluded old baseline, a matching tag and its descendants, and an unreachable tag.
The key review question is: Does the custom fallback reliably get out of the way once standard Git-derived versioning can work?
The durable operating procedure is RELEASE-bindings.md, with shared ownership and rollover guidance in ci/README.md. The contributor CI overview now uses a maintained conceptual Mermaid diagram.
Implementation and Review Guide
12.9.xbackport policy. Adds explicit manual automation for selected stabilization branches.Refreshed six-layer review aid
Read the new review-only branch in the order below, or open all six commits and the complete diff.
The branch starts at main commit
ab9e20fd6a525d526d85d529571b3940824e8467. Its final commit has exactly the same Git tree (6e95a59f35e36ab401e97eab785ae8f8253fa2eb) as the tested #2737 head6c3de41b319622453b1a8ca38f21166c79db1a3a. Git verification confirms six single-parent commits and all 302 changed paths assigned once.These are reading layers: each shared file is assigned in full to one layer, and intermediate commits need not build independently. The final tree is identical, but the reconstructed history differs; validation and merge decisions belong to #2737. This replaces the earlier review aid #2960, which remains a historical snapshot.
Validation
Current implementation head:
6c3de41b319622453b1a8ca38f21166c79db1a3a.main/cuda_bindings_12URL failures. Hosted pre-commit on both platforms is green.6c3de41b319622453b1a8ca38f21166c79db1a3a(attempt 2). Pixi source-build/docs and lockfile freshness checks, Bandit, and the security suite are also green. In attempt 1, the Linux Python 3.14t / CUDA 12.9.1 / T4 job timed out after 90 minutes in a CuPy interop test; the attempt was then cancelled while two Windows L4 jobs were still queued, allowing a retry of the unfinished jobs. All three passed on the unchanged commit. The Linux retry reports 4,390 passing Core tests, with the previously stalled test passing in parallel in 21.6 seconds. The initial timeout's cause remains undiagnosed.pyyaml-6.0.3-py3-none-any.whl, followed by 4,092 passing Core tests. All four Windows Python 3.15/3.15t native builds also pass. All Python 3.15 checks are included in this reported validation.Earlier CUDA 12.9 local validation passed 390 maintenance bindings tests in each mode and its Cython tests. Follow-up cuFile isolation fixes restore global parameters and serialize the CUDA 12 module's process-global driver lifecycle; its parallel-run regression check passed 63 cases. These maintenance-runtime results precede the current workflow and metadata-validation fixes.
Earlier release rehearsals and their limits
The September 29 CUDA 12 package-source rehearsal passed all 48 build, sdist, GPU-test, documentation, and prerequisite jobs at source
b00b677be366fd3638c78e26b99aa5b238602646. The original run failed its final custom harness assertion. Its validation follow-up passed, checking production wheel validation, package metadata, artifact provenance, and exact-source archive identity after correcting a harness expectation. The synthetic tag stayed local, with publication and docs deployment disabled. These results describe an earlier source revision.Four earlier dry runs exercised retained real-tag artifacts: 13.4.3 bindings, 13.4.3 metapackage, 12.9.9 bindings, and 12.9.9 metapackage. They used the compatibility code subsequently removed from this PR. They are historical development evidence; the current tooling deliberately rejects these pre-registry tags.
Production release validation still requires a supported immutable tag and successful exact-tag/exact-SHA CI. Branch CI and synthetic rehearsals do not authorize publication.
Remaining Boundaries
The stale generated-file seal inherited from #2911 was corrected in 548b2615, with trailing blank-line normalization documented. This does not establish a fresh cybind regeneration.
Inherited documentation boundaries remain: two NVRTC generated-doc entries reference functions absent from the maintenance module, and README links retain the published 12.9.7 documentation. Archive digest companions retain their existing raw-digest format. After merge, recheck the three new canonical
main/cuda_bindings_12links that are unavailable before the directory reachesmain.Additional simultaneous public roots, unreleased-toolkit qualification, source-tree deduplication, and private QA reorganization remain outside this change.
Merge Status
The refreshed six-layer review aid matches the tested implementation head. The Fable 5 re-review confirms all three follow-up fixes and reports no new findings; its listed minor items remain follow-up work. The existing changes-requested review remains for renewed reviewer consideration. Production tags and publication require the normal separate release approval.
Author-side review disposition map (45 initially open threads)
ci/versions.ymlin the tagged source. Reject pre-registry layouts; identify artifacts by exact tag and SHA, validate the complete wheel matrix, exclude test wheels, and require explicit dependency sources. Historical compatibility was removed in the October 4 design simplification.These dispositions describe the implementation and retained design choices. They do not represent reviewer approval.