Repository navigation
Speed up file installation on Windows - #3832
Conversation
|
FYI I ran the "Test: critical" integration tests to make sure this isn't breaking anything unexpected. The failures that did occur are sporadic and unrelated to this PR. |
| // artifacts can be linked into the runtime concurrently (the install worker pool runs several at once). | ||
| // This is safe because linking touches no shared in-memory depot state (d.Exists guards its own access | ||
| // via mapMutex, and d.depotPath is immutable after construction), directory creation is idempotent, and | ||
| // smartlink treats an already-existing target as a skip. This is a major win on Windows where each file |
There was a problem hiding this comment.
I am not terribly comfortable with this assertion. It's probably referring to this code in smartlink.go:
// Multiple artifacts can supply the same file. We do not have a better solution for this at the moment other than
// favouring the first one encountered.
if fileutils.TargetExists(dest) {
logging.Warning("Skipping linking '%s' to '%s' as it already exists", src, dest)
return nil
}If we remove the fs mutex and then come up with a better solution, we lose the mutex guarantee. If we want to remove the mutex, we should update the referenced code's comment to mention the lack of an existing mutex (the one we're removing).
I like the additional comment above DeployViaCopy that we need the mutex to avoid copying to the same destination, but I'm not convinced we can forgo the need for mutex when creating links.
There was a problem hiding this comment.
We could try making fsMutex an RWMutex and have DeployViaLink take an RLock and DeployViaCopy/Put/Undeploy take Lock. This way the safer operations which are linking can be done in parallel but copying which isn't as safe still has to be sequential.
There was a problem hiding this comment.
You were right not to trust this. Removing the lock outright was unsafe: a copy deployment checks that a destination is free and then calls os.Create. If a concurrent link put a link to another artifact's depot file there in between, the copy wrote through it and overwrote the depot file. TestConcurrentCopyAndLinkKeepDepotIntact reproduces that in 10/10 runs without the lock.
I've gone with the RWMutex suggestion (365652c): DeployViaLink takes RLock, and DeployViaCopy, Put and Undeploy take Lock. Concurrent links only race with each other, which is safe: creating a link never writes through an existing file, and an "already exists" error from the link call is treated as the existing first-one-wins skip. The shared-file comment in smartlink.go now says that artifacts may be linked concurrently and that the caller must not copy at the same time, so a future change there can see what it relies on.
The read lock costs nothing measurable on Windows (10.2 s vs 10.4 s without any lock); the benchmark table is in the PR description.
MDrakos
left a comment
There was a problem hiding this comment.
This PR needs some tests, especially to test the concurrency element which this PR addresses.
Some ideas from my robot all run with -race:
- A deterministic unit test for the new os.IsExist branch: pre-create the target,
call Link, assert it returns nil and leaves the existing target intact. - A concurrent same-file link test in smartlink (two goroutines, same dest) to hit
the TargetExists -> linkFile -> os.IsExist path. - If we take the RWMutex: a pkg/runtime test deploying a copy artifact and a link
artifact that share a file concurrently, asserting the depot source stays
byte-for-byte identical — the regression guard for the truncation case. - A quick equivalence test for the Link refactor (regular dir / symlink->file /
symlink->dir), since only the circular case is covered today.
| // artifacts can be linked into the runtime concurrently (the install worker pool runs several at once). | ||
| // This is safe because linking touches no shared in-memory depot state (d.Exists guards its own access | ||
| // via mapMutex, and d.depotPath is immutable after construction), directory creation is idempotent, and | ||
| // smartlink treats an already-existing target as a skip. This is a major win on Windows where each file |
There was a problem hiding this comment.
We could try making fsMutex an RWMutex and have DeployViaLink take an RLock and DeployViaCopy/Put/Undeploy take Lock. This way the safer operations which are linking can be done in parallel but copying which isn't as safe still has to be sequential.
smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Linking without the depot lock let a concurrent copy deployment write through a link created moments earlier: copyFiles checks the destination is free, a link to another artifact's depot file appears there, and os.Create then truncates and overwrites that depot file, corrupting the shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in every run without the lock. Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links still run concurrently, while copies, Put and Undeploy hold it for writing. Also fix the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole package. Add smartlink tests for an existing target, concurrent links to the same destination, and the Link refactor's handling of directories and symlinks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its parent and TargetExists before linking it, plus another IsDir inside the Windows linkFile. Each is a file open on Windows. Instead: - take the entry type from the directory listing (os.ReadDir already has it), so only symlinks need a Stat; - rely on the parent directory existing, since the recursion creates it (Link creates it for its single entry); - drop the TargetExists pre-check and treat the link call's "already exists" error as the skip, which already covered concurrent links; - check for a directory only on the symlink-to-directory path, which is the only one that can pass one to linkFile. A regular file now costs one syscall: the link itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1192e70 to
a361a29
Compare
|
@MDrakos thanks for the list. All four tests are added, and the race checks pass under
I also benchmarked the change natively on Windows. With a further commit that drops redundant per-file metadata calls, linking a 22k-file runtime went from 75 s to 7 s; details are in the updated description. |
* Resolve link paths once per tree instead of once per file smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Deploy linked artifacts concurrently instead of serializing them DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Hold a read lock while linking so copies can't write through links Linking without the depot lock let a concurrent copy deployment write through a link created moments earlier: copyFiles checks the destination is free, a link to another artifact's depot file appears there, and os.Create then truncates and overwrites that depot file, corrupting the shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in every run without the lock. Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links still run concurrently, while copies, Put and Undeploy hold it for writing. Also fix the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole package. Add smartlink tests for an existing target, concurrent links to the same destination, and the Link refactor's handling of directories and symlinks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip per-file metadata calls when linking link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its parent and TargetExists before linking it, plus another IsDir inside the Windows linkFile. Each is a file open on Windows. Instead: - take the entry type from the directory listing (os.ReadDir already has it), so only symlinks need a Stat; - rely on the parent directory existing, since the recursion creates it (Link creates it for its single entry); - drop the TargetExists pre-check and treat the link call's "already exists" error as the skip, which already covered concurrent links; - check for a directory only on the symlink-to-directory path, which is the only one that can pass one to linkFile. A regular file now costs one syscall: the link itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Resolve link paths once per tree instead of once per file smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Deploy linked artifacts concurrently instead of serializing them DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Hold a read lock while linking so copies can't write through links Linking without the depot lock let a concurrent copy deployment write through a link created moments earlier: copyFiles checks the destination is free, a link to another artifact's depot file appears there, and os.Create then truncates and overwrites that depot file, corrupting the shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in every run without the lock. Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links still run concurrently, while copies, Put and Undeploy hold it for writing. Also fix the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole package. Add smartlink tests for an existing target, concurrent links to the same destination, and the Link refactor's handling of directories and symlinks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip per-file metadata calls when linking link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its parent and TargetExists before linking it, plus another IsDir inside the Windows linkFile. Each is a file open on Windows. Instead: - take the entry type from the directory listing (os.ReadDir already has it), so only symlinks need a Stat; - rely on the parent directory existing, since the recursion creates it (Link creates it for its single entry); - drop the TargetExists pre-check and treat the link call's "already exists" error as the skip, which already covered concurrent links; - check for a directory only on the symlink-to-directory path, which is the only one that can pass one to linkFile. A regular file now costs one syscall: the link itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Speed up file installation on Windows (#3832) * Resolve link paths once per tree instead of once per file smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Deploy linked artifacts concurrently instead of serializing them DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Hold a read lock while linking so copies can't write through links Linking without the depot lock let a concurrent copy deployment write through a link created moments earlier: copyFiles checks the destination is free, a link to another artifact's depot file appears there, and os.Create then truncates and overwrites that depot file, corrupting the shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in every run without the lock. Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links still run concurrently, while copies, Put and Undeploy hold it for writing. Also fix the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole package. Add smartlink tests for an existing target, concurrent links to the same destination, and the Link refactor's handling of directories and symlinks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip per-file metadata calls when linking link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its parent and TargetExists before linking it, plus another IsDir inside the Windows linkFile. Each is a file open on Windows. Instead: - take the entry type from the directory listing (os.ReadDir already has it), so only symlinks need a Stat; - rely on the parent directory existing, since the recursion creates it (Link creates it for its single entry); - drop the TargetExists pre-check and treat the link call's "already exists" error as the skip, which already covered concurrent links; - check for a directory only on the symlink-to-directory path, which is the only one that can pass one to linkFile. A regular file now costs one syscall: the link itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 8f473b3) * Remediate CVE-2026-63209 in klauspost/compress (#3858) Bump github.com/klauspost/compress (indirect, via mholt/archives) from v1.18.0 to v1.18.7, the first patched release for GHSA-259r-337f-4rfw (integer overflow and out-of-bounds write in s2). Re-vendored. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b09f7ee) * Support dependencies for encrypted private ingredients (#3857) * Support dependencies for encrypted private ingredients `state publish --build` can now declare and edit the dependencies of an encrypted private ingredient: - Read [project].dependencies from pyproject.toml and publish them as runtime dependencies on language/python packages, with names normalized by the inventory API. Version ranges carry over (~= and ==X.* are expanded). OS and Python version markers become conditions (kernel/Windows|Linux|Darwin, language/python); unsupported markers are skipped with a notice. The wheel's METADATA records them as Requires-Dist. - Read the --meta file before building, so a meta file's name, version and description feed the wheel (flags > meta file > pyproject.toml). - Add `state export ingredient-meta <namespace>/<name>[@Version]`, which writes an existing ingredient's meta file for use with --edit --meta. Fix --edit along the way: keep the meta file's dependencies and authors instead of overwriting them with the previous revision's, keep inherited dependency types and conditions, add the private-builder dependency only once, and reject author/description changes on an existing version up front instead of failing in the API. InventoryRequirementsToString no longer panics on requirements without a version. Add a readme walking through the publish and edit workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: keep inherited deps, edit the meta file's version - When every pyproject.toml dependency is skipped, keep the dependencies already in place (on --edit, those inherited from the previous revision) instead of replacing them with an empty list. - Let a meta file's version select the version to publish or edit, like --version (which still takes precedence). An exported meta file passed back with `state publish --edit --meta` without --build was editing the latest version instead of the exported one. Add tests for both, for the up-front author/description rejection on --edit, for InventoryRequirementsToString with requirements missing a version or comparator, and for inherited conditions that carry requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 44e750f) * Add 0.48.2 changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update version.txt * 0.48.2-RC1: #3832, #3858, #3857 on top of 0.48.1-RC4 (#3863) * Speed up file installation on Windows (#3832) * Resolve link paths once per tree instead of once per file smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Deploy linked artifacts concurrently instead of serializing them DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Hold a read lock while linking so copies can't write through links Linking without the depot lock let a concurrent copy deployment write through a link created moments earlier: copyFiles checks the destination is free, a link to another artifact's depot file appears there, and os.Create then truncates and overwrites that depot file, corrupting the shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in every run without the lock. Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links still run concurrently, while copies, Put and Undeploy hold it for writing. Also fix the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole package. Add smartlink tests for an existing target, concurrent links to the same destination, and the Link refactor's handling of directories and symlinks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip per-file metadata calls when linking link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its parent and TargetExists before linking it, plus another IsDir inside the Windows linkFile. Each is a file open on Windows. Instead: - take the entry type from the directory listing (os.ReadDir already has it), so only symlinks need a Stat; - rely on the parent directory existing, since the recursion creates it (Link creates it for its single entry); - drop the TargetExists pre-check and treat the link call's "already exists" error as the skip, which already covered concurrent links; - check for a directory only on the symlink-to-directory path, which is the only one that can pass one to linkFile. A regular file now costs one syscall: the link itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 8f473b3) * Remediate CVE-2026-63209 in klauspost/compress (#3858) Bump github.com/klauspost/compress (indirect, via mholt/archives) from v1.18.0 to v1.18.7, the first patched release for GHSA-259r-337f-4rfw (integer overflow and out-of-bounds write in s2). Re-vendored. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b09f7ee) * Support dependencies for encrypted private ingredients (#3857) * Support dependencies for encrypted private ingredients `state publish --build` can now declare and edit the dependencies of an encrypted private ingredient: - Read [project].dependencies from pyproject.toml and publish them as runtime dependencies on language/python packages, with names normalized by the inventory API. Version ranges carry over (~= and ==X.* are expanded). OS and Python version markers become conditions (kernel/Windows|Linux|Darwin, language/python); unsupported markers are skipped with a notice. The wheel's METADATA records them as Requires-Dist. - Read the --meta file before building, so a meta file's name, version and description feed the wheel (flags > meta file > pyproject.toml). - Add `state export ingredient-meta <namespace>/<name>[@Version]`, which writes an existing ingredient's meta file for use with --edit --meta. Fix --edit along the way: keep the meta file's dependencies and authors instead of overwriting them with the previous revision's, keep inherited dependency types and conditions, add the private-builder dependency only once, and reject author/description changes on an existing version up front instead of failing in the API. InventoryRequirementsToString no longer panics on requirements without a version. Add a readme walking through the publish and edit workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: keep inherited deps, edit the meta file's version - When every pyproject.toml dependency is skipped, keep the dependencies already in place (on --edit, those inherited from the previous revision) instead of replacing them with an empty list. - Let a meta file's version select the version to publish or edit, like --version (which still takes precedence). An exported meta file passed back with `state publish --edit --meta` without --build was editing the latest version instead of the exported one. Add tests for both, for the up-front author/description rejection on --edit, for InventoryRequirementsToString with requirements missing a version or comparator, and for inherited conditions that carry requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 44e750f) * Add 0.48.2 changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * check-format: don't fail when a PR changes no files. With an empty diff, grep -v exits 1 and set -e aborts the script with no output, so a PR that changes nothing (e.g. merging beta into a version branch) fails Check Format on every OS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Update to Go 1.26.10 and golang.org/x/net v0.60.0 to remediate CVEs. The CI Trivy scan fails on 13 Go stdlib CVEs (fixed in 1.26.9/1.26.10) and 11 golang.org/x/net findings (fixed in v0.60.0) published since the 0.48.1 scan. Re-vendored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Compare RSA keys by value in keypair tests. The tests compared a generated keypair with the same key parsed back from PEM using deep struct equality, which also compares rsa.PrivateKey's precomputed internals. Under Go 1.26.10 those can differ (e.g. a leading zero byte in dP) even though the keys are identical, so the tests failed intermittently. Compare with rsa.PrivateKey.Equal instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Allow for slow sw_vers in the macOS OSVersion performance test. (#3866) TestPerformance/OSVersion averages ten calls against a 10ms budget, so the single uncached call bounds it at ~100ms. On macOS that call shells out to sw_vers, which took 115-716ms on macos-latest runners and failed 3 of 4 runs. Cached calls take nanoseconds. Keep 10ms elsewhere; allow 100ms on darwin. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: mitchell <mitchellb@activestate.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* 0.48.2-RC1 (#3864) * Update version.txt * 0.48.2-RC1: #3832, #3858, #3857 on top of 0.48.1-RC4 (#3863) * Speed up file installation on Windows (#3832) * Resolve link paths once per tree instead of once per file smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Deploy linked artifacts concurrently instead of serializing them DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Hold a read lock while linking so copies can't write through links Linking without the depot lock let a concurrent copy deployment write through a link created moments earlier: copyFiles checks the destination is free, a link to another artifact's depot file appears there, and os.Create then truncates and overwrites that depot file, corrupting the shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in every run without the lock. Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links still run concurrently, while copies, Put and Undeploy hold it for writing. Also fix the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole package. Add smartlink tests for an existing target, concurrent links to the same destination, and the Link refactor's handling of directories and symlinks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip per-file metadata calls when linking link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its parent and TargetExists before linking it, plus another IsDir inside the Windows linkFile. Each is a file open on Windows. Instead: - take the entry type from the directory listing (os.ReadDir already has it), so only symlinks need a Stat; - rely on the parent directory existing, since the recursion creates it (Link creates it for its single entry); - drop the TargetExists pre-check and treat the link call's "already exists" error as the skip, which already covered concurrent links; - check for a directory only on the symlink-to-directory path, which is the only one that can pass one to linkFile. A regular file now costs one syscall: the link itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 8f473b3) * Remediate CVE-2026-63209 in klauspost/compress (#3858) Bump github.com/klauspost/compress (indirect, via mholt/archives) from v1.18.0 to v1.18.7, the first patched release for GHSA-259r-337f-4rfw (integer overflow and out-of-bounds write in s2). Re-vendored. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b09f7ee) * Support dependencies for encrypted private ingredients (#3857) * Support dependencies for encrypted private ingredients `state publish --build` can now declare and edit the dependencies of an encrypted private ingredient: - Read [project].dependencies from pyproject.toml and publish them as runtime dependencies on language/python packages, with names normalized by the inventory API. Version ranges carry over (~= and ==X.* are expanded). OS and Python version markers become conditions (kernel/Windows|Linux|Darwin, language/python); unsupported markers are skipped with a notice. The wheel's METADATA records them as Requires-Dist. - Read the --meta file before building, so a meta file's name, version and description feed the wheel (flags > meta file > pyproject.toml). - Add `state export ingredient-meta <namespace>/<name>[@Version]`, which writes an existing ingredient's meta file for use with --edit --meta. Fix --edit along the way: keep the meta file's dependencies and authors instead of overwriting them with the previous revision's, keep inherited dependency types and conditions, add the private-builder dependency only once, and reject author/description changes on an existing version up front instead of failing in the API. InventoryRequirementsToString no longer panics on requirements without a version. Add a readme walking through the publish and edit workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: keep inherited deps, edit the meta file's version - When every pyproject.toml dependency is skipped, keep the dependencies already in place (on --edit, those inherited from the previous revision) instead of replacing them with an empty list. - Let a meta file's version select the version to publish or edit, like --version (which still takes precedence). An exported meta file passed back with `state publish --edit --meta` without --build was editing the latest version instead of the exported one. Add tests for both, for the up-front author/description rejection on --edit, for InventoryRequirementsToString with requirements missing a version or comparator, and for inherited conditions that carry requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 44e750f) * Add 0.48.2 changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * check-format: don't fail when a PR changes no files. With an empty diff, grep -v exits 1 and set -e aborts the script with no output, so a PR that changes nothing (e.g. merging beta into a version branch) fails Check Format on every OS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Update to Go 1.26.10 and golang.org/x/net v0.60.0 to remediate CVEs. The CI Trivy scan fails on 13 Go stdlib CVEs (fixed in 1.26.9/1.26.10) and 11 golang.org/x/net findings (fixed in v0.60.0) published since the 0.48.1 scan. Re-vendored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Compare RSA keys by value in keypair tests. The tests compared a generated keypair with the same key parsed back from PEM using deep struct equality, which also compares rsa.PrivateKey's precomputed internals. Under Go 1.26.10 those can differ (e.g. a leading zero byte in dP) even though the keys are identical, so the tests failed intermittently. Compare with rsa.PrivateKey.Equal instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Allow for slow sw_vers in the macOS OSVersion performance test. (#3866) TestPerformance/OSVersion averages ten calls against a 10ms budget, so the single uncached call bounds it at ~100ms. On macOS that call shells out to sw_vers, which took 115-716ms on macos-latest runners and failed 3 of 4 runs. Cached calls take nanoseconds. Keep 10ms elsewhere; allow 100ms on darwin. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: mitchell <mitchellb@activestate.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * 0.48.2-RC2 (#3870) * Update version.txt * Fix state publish meta data issues found testing 0.48.2-RC1 (#3867) * Fix state publish meta data issues found testing 0.48.2-RC1 - state export ingredient-meta writes plain yaml, unwrapped and without field help comments, so the file can be passed back with --meta. - Dependencies and conditions without a version range (a bare name in pyproject.toml, --depend without a version, or a condition inherited without requirements) get >=0. The platform rejected them with "Expecting an OBJECT with a field named 'versionRequirements'". - When --build can't determine the name or version, say which is missing, why pyproject.toml didn't provide it (no file, not in [project], or listed in [project].dynamic), and how to set it. - Answering 'n' at the publish confirmation offers to edit the meta data in the editor or save it to <name>-<version>.meta.yaml with the command to publish it, instead of only cancelling. With --build or --edit, a name, namespace or version change in the editor is saved to a file instead, as the package was already built and looked up with them. - Fields removed in the editor are removed from the request, and the publish API error is shown instead of <no value>. The platform requires the source with every publish, so --edit still needs --build or an archive; the readme now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Update echo to v4.16.1 and drop an unused publish error argument echo v4.15.4 has GHSA-cxf3-9h37-7rr8 (critical), GHSA-9969-6676-49h5, GHSA-f48p-c9v4-h7xv and GHSA-h7wr-284f-5qvc, which failed the Scan job. err_uploadingredient_publish is defined in en-us.yaml without a {{.V0}}, so the joined message passed to it was never shown. The wrapped error is already printed beneath it, so drop the argument and the changelog claim. * Keep the API error in the publish failure message err_uploadingredient_publish in en-us.yaml ends in "Error received: {{.V0}}" and overrides the fallback text, so without the argument the message reads "<no value>". The wrapped API error isn't user-facing, so it isn't printed beneath it either; it only reaches the log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix TestOpen failing on Windows; correct the publish error comment TestOpen removed its temp file while the hail watcher could still be reading it for a later write event: the remove was deferred after the cancel, so it ran first. On Windows an open file can't be removed. Stop the watcher and drain its channel before removing the file. The publish error comment said the wrapped API error is never printed; a transport failure becomes a user-facing BuildPlannerError and is. Only an error reported in the publish result (an errs.New) relies on {{.V0}}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Validate each confirm pass and test the publish fixes through their call paths Address review on #3867: - Run the namespace and checkEditKeepsIdentity checks at the top of each pass of the confirm loop, now confirmRequest, so a namespace cleared or a description added in the editor is rejected with a clear message instead of failing in MarshalJSON or the API. - Add tests that fail if a fix is reverted: confirmRequest with a fake prompter (>=0 filling, editor removing fields, cleared namespace, description on a revision, private-builder restored, save, cancel), using the test binary as $EDITOR so it also runs on Windows; publishFailedError keeping the API error; metaFromIngredient filling conditions; and writeIngredientMeta writing plain yaml that round-trips. FetchMeta's pure part moves into metaFromIngredient and the export's writing into writeIngredientMeta so they can be tested without the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Fix CI for 0.48.2-RC2: x/net CVEs, publish --editor, and runner-specific test failures (#3869) * Update golang.org/x/net to v0.62.0 for CVE-2026-78661 and CVE-2026-97045 The Scan job failed on both: x/net v0.60.0 is affected and v0.62.0 fixes them. go get also raised x/crypto, x/sys, x/term, x/text, x/sync and x/tools to the versions x/net v0.62.0 requires. * Don't search every namespace when publishing a new ingredient without one state publish --editor outside a project has no namespace until the editor fills it in, so Run searched every namespace for the name derived from the file. Logged in, that search took long enough to time out ("Could not search for ingredient"), so the editor never opened. This is why the "New ingredient with editor flag" integration test has failed nightly on master since at least September. Its first match could also be any ingredient with that name. Skip the lookup for a new ingredient without a namespace; confirmRequest requires one and checkNamespaceEdit checks the version once it is filled in. --edit still searches, since it needs to find the ingredient. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let state publish --editor continue when no editor can be opened Without EDITOR set and without xdg-open, as on many headless Linux machines and the ubuntu-24.04-arm runner, OpenEditor fails and publish exited with "Could not publish ingredient. Error received: <no value>", even though the notice just before it says the file can be edited by hand. Say that no editor could be opened and wait for the confirmation as usual instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix integration tests that fail only because of the runner or branch name - TestPublishBuildEncrypted installs into a Python project that state init creates without a Linux/arm64 platform, so skip it there, like the other platform-specific skips. Integration tests only run on ARM for labelled PRs. - PerformanceIntegrationTestSuite's TearDownSuite stopped state-svc even when its tests were skipped by tag. On macOS a long branch channel makes the socket path too long to dial, so every labelled PR from such a branch failed there. Only stop the service when the Performance tests ran, using a new tagsuite.RunsForTags that OnlyRunForTags is now built on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Don't open a real editor in the publish --editor integration test The test writes its edits to the file itself, but with no EDITOR set, macOS `open` hands the file to whatever app the runner has registered. On macos-latest, publish then read the file back without the test's edits and failed with "You have to supply the namespace". Point EDITOR at a command that doesn't exist, so publish takes the "edit the file by hand" path on every OS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix the publish --editor integration test's flaky file path match editorFileRx matched "file:\s*?(.*?)\.\s", whose lazy \s*? left the space after "file:" in the captured path whenever the temp file name had 9 digits instead of 10 (about one run in four). The test then wrote its edits to " /tmp/NNN.ingredient.yaml", a different file, so publish read the original back and failed with "You have to supply the namespace". The last commit's macOS explanation was wrong; this was the cause on every OS. Reproduced by driving a dev build through termtest 12 times: three runs captured the leading space before this change and none after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Make the state publish meta data clearer to review and edit (#3868) * Update golang.org/x/net to v0.62.0 for CVE-2026-78661 and CVE-2026-97045 The Scan job failed on both: x/net v0.60.0 is affected and v0.62.0 fixes them. go get also raised x/crypto, x/sys, x/term, x/text, x/sync and x/tools to the versions x/net v0.62.0 requires. * Don't search every namespace when publishing a new ingredient without one state publish --editor outside a project has no namespace until the editor fills it in, so Run searched every namespace for the name derived from the file. Logged in, that search took long enough to time out ("Could not search for ingredient"), so the editor never opened. This is why the "New ingredient with editor flag" integration test has failed nightly on master since at least September. Its first match could also be any ingredient with that name. Skip the lookup for a new ingredient without a namespace; confirmRequest requires one and checkNamespaceEdit checks the version once it is filled in. --edit still searches, since it needs to find the ingredient. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let state publish --editor continue when no editor can be opened Without EDITOR set and without xdg-open, as on many headless Linux machines and the ubuntu-24.04-arm runner, OpenEditor fails and publish exited with "Could not publish ingredient. Error received: <no value>", even though the notice just before it says the file can be edited by hand. Say that no editor could be opened and wait for the confirmation as usual instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix integration tests that fail only because of the runner or branch name - TestPublishBuildEncrypted installs into a Python project that state init creates without a Linux/arm64 platform, so skip it there, like the other platform-specific skips. Integration tests only run on ARM for labelled PRs. - PerformanceIntegrationTestSuite's TearDownSuite stopped state-svc even when its tests were skipped by tag. On macOS a long branch channel makes the socket path too long to dial, so every labelled PR from such a branch failed there. Only stop the service when the Performance tests ran, using a new tagsuite.RunsForTags that OnlyRunForTags is now built on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Don't open a real editor in the publish --editor integration test The test writes its edits to the file itself, but with no EDITOR set, macOS `open` hands the file to whatever app the runner has registered. On macos-latest, publish then read the file back without the test's edits and failed with "You have to supply the namespace". Point EDITOR at a command that doesn't exist, so publish takes the "edit the file by hand" path on every OS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix the publish --editor integration test's flaky file path match editorFileRx matched "file:\s*?(.*?)\.\s", whose lazy \s*? left the space after "file:" in the captured path whenever the temp file name had 9 digits instead of 10 (about one run in four). The test then wrote its edits to " /tmp/NNN.ingredient.yaml", a different file, so publish read the original back and failed with "You have to supply the namespace". The last commit's macOS explanation was wrong; this was the cause on every OS. Reproduced by driving a dev build through termtest 12 times: three runs captured the leading space before this change and none after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make the state publish meta data clearer to review and edit Follow-up to #3867 from testing the 0.48.2-RC2 build: - Without --namespace, a --build publish from a pyproject.toml now defaults to private/<org>/python (otherwise private/<org>, as before). - A new ingredient's namespace can be changed in the editor. It is checked like --namespace and put back, with the reason, if it can't be used. - The confirmation and editor say up front which fields can't be changed and why, and the error for changing one names it. - The preview and editor show plain yaml without a help comment on every field; the fields are described in the publish guide instead. The yaml-comment module is no longer used and is removed. - state export ingredient-meta now includes the ingredient's description. state publish --edit no longer refuses a different description on an existing version: it keeps the current one and says the change is ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: mitchell <mitchellb@activestate.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Why
Installing runtimes is slow on Windows, and the deploy step (linking depot files into the runtime directory) is a large part of it. Timing it on Windows with a real runtime (32 artifacts, 22,456 files) showed
mastertakes about 75 s to link, roughly 3.4 ms per file. The cost was almost entirely per-file metadata syscalls, plus the depot lock serializing artifacts.Changes
Resolve link paths once per tree instead of once per file (
smartlink.go).Linkre-resolved src and dest withResolveUniquePathfor every entry. On Windows that isEvalSymlinks(anLstatper path component) plusGetLongPathName. The recursion now resolves the root once and builds descendant paths from resolved parents and real entry names.Deploy linked artifacts concurrently (
depot.go).DeployViaLinkno longer holds the depot lock exclusively, so the install worker pool links several artifacts at once.Hold a read lock while linking. This addresses the review feedback. Fully removing the lock (as in the first revision) was unsafe: a copy deployment checks that a destination is free and then calls
os.Create. If a concurrent link created a link to another artifact's depot file in between, the copy truncated and overwrote that depot file, corrupting the shared cache. The newTestConcurrentCopyAndLinkKeepDepotIntactreproduces this in 10/10 runs without the lock (on any OS; on Linux/macOS the links are symlinks, whichos.Createfollows).fsMutexis now anRWMutex:DeployViaLinktakesRLock;DeployViaCopy,PutandUndeploytakeLock. Concurrent links are safe with each other: creating a link never writes through an existing file, directory creation is idempotent, and an "already exists" error from the link call is treated as the existing first-one-wins skip. The shared-file comment insmartlinknow says what it relies on. This commit also fixes the non-constant format string inDeployViaCopy, which madego test's vet step fail the wholepkg/runtimepackage.Skip per-file metadata calls when linking (
smartlink.go,smartlink_windows.go). Each file still went throughIsDir,IsSymlink,MkdirUnlessExistson its parent,TargetExists, and anotherIsDirin the WindowslinkFile. Now:os.ReadDir, so only symlinks need aStat;TargetExistspre-check;A regular file now costs one syscall: the link.
Results
Deploying the same 32 artifacts via
DeployViaLinkwith a 5-worker pool (as insetup.update), measured natively on Windows 10 (12 logical CPUs). Variants were interleaved, with a warm-up plus 4–6 timed runs each.mastermastermasterwithout the lock (concurrency only)The read lock costs nothing measurable compared with removing the lock.
Caveats:
state checkout.Tests
The tests asked for in review:
TestConcurrentCopyAndLinkKeepDepotIntact(pkg/runtime) deploys a copy artifact and a link artifact that share 200 files into one runtime concurrently, 20 times, and checks both depot sources are byte-for-byte intact. It fails 10/10 on the first revision of this PR and passes onmasterand on this PR (20/20 under-race).TestLinkContentsConcurrentSameDestlinks 8 sources providing the same files into one destination concurrently. It fails onmasterwith "file exists" and passes here.TestLinkSkipsExistingTargetchecks that an existing target is left intact.TestLinkContentsEquivalencecovers regular directories, a symlink to a file and a symlink to a directory. It passes onmaster's implementation too, so the refactor doesn't change linking behavior.go test -race ./internal/smartlink/ ./pkg/runtime/passes on Linux. On Windows the new smartlink tests pass. The existingTestLinkContentsWithCircularLinkfails on the benchmark machine both here and onmaster, because creating its symlink fixture needs symlink privileges ("A required privilege is not held by the client").Not addressed (follow-ups)
ApplyFileTransformsruns afterDeployViaCopy, outside the lock, and writes files in place. If another artifact's link already occupies a transformed path (copy skipped it as existing), the transform writes into that artifact's depot file. This can happen regardless of concurrency, depending on install order. A fix is to have transforms replace the file (write and rename) rather than write in place.🤖 Generated with Claude Code