Skip to content

Speed up file installation on Windows - #3832

Merged
icanhasmath merged 4 commits into
masterfrom
windows-install-perf
Oct 21, 2026
Merged

icanhasmath merged 4 commits into
masterfrom
windows-install-perf

Conversation

@icanhasmath

@icanhasmath icanhasmath commented Jul 6, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Installing runtimes is slow on Windows, and the deploy step (linking depot files into the runtime directory) is a large part of it. Timing it on Windows with a real runtime (32 artifacts, 22,456 files) showed master takes about 75 s to link, roughly 3.4 ms per file. The cost was almost entirely per-file metadata syscalls, plus the depot lock serializing artifacts.

Changes

  1. Resolve link paths once per tree instead of once per file (smartlink.go). Link re-resolved src and dest with ResolveUniquePath for every entry. On Windows that is EvalSymlinks (an Lstat per path component) plus GetLongPathName. The recursion now resolves the root once and builds descendant paths from resolved parents and real entry names.

  2. Deploy linked artifacts concurrently (depot.go). DeployViaLink no longer holds the depot lock exclusively, so the install worker pool links several artifacts at once.

  3. Hold a read lock while linking. This addresses the review feedback. Fully removing the lock (as in the first revision) was unsafe: a copy deployment checks that a destination is free and then calls os.Create. If a concurrent link created a link to another artifact's depot file in between, the copy truncated and overwrote that depot file, corrupting the shared cache. The new TestConcurrentCopyAndLinkKeepDepotIntact reproduces this in 10/10 runs without the lock (on any OS; on Linux/macOS the links are symlinks, which os.Create follows). fsMutex is now an RWMutex: DeployViaLink takes RLock; DeployViaCopy, Put and Undeploy take Lock. Concurrent links are safe with each other: creating a link never writes through an existing file, directory creation is idempotent, and an "already exists" error from the link call is treated as the existing first-one-wins skip. The shared-file comment in smartlink now says what it relies on. This commit also fixes the non-constant format string in DeployViaCopy, which made go test's vet step fail the whole pkg/runtime package.

  4. Skip per-file metadata calls when linking (smartlink.go, smartlink_windows.go). Each file still went through IsDir, IsSymlink, MkdirUnlessExists on its parent, TargetExists, and another IsDir in the Windows linkFile. Now:

    • the entry type comes from os.ReadDir, so only symlinks need a Stat;
    • the parent directory is known to exist from the recursion;
    • the "already exists" error replaces the TargetExists pre-check;
    • the directory check runs only on the symlink-to-directory path.

    A regular file now costs one syscall: the link.

Results

Deploying the same 32 artifacts via DeployViaLink with a 5-worker pool (as in setup.update), measured natively on Windows 10 (12 logical CPUs). Variants were interleaved, with a warm-up plus 4–6 timed runs each.

Variant Median vs master
master 75.4 s 1.0×
master without the lock (concurrency only) 30.1 s 2.5×
Commit 1 only (resolve once) 21.9 s 3.4×
Commits 1–2 (first revision of this PR, unsafe) 10.4 s 7.3×
Commits 1–3 (read lock) 10.2 s 7.4×
Commits 1–4 (this PR) 7.0 s 10.7×

The read lock costs nothing measurable compared with removing the lock.

Caveats:

  • Defender real-time protection was off on the benchmark machine. Defender adds latency to every file open, so the absolute savings are likely larger on typical machines, but that wasn't measured.
  • The numbers cover the link step on its own, not a full state checkout.

Tests

The tests asked for in review:

  • TestConcurrentCopyAndLinkKeepDepotIntact (pkg/runtime) deploys a copy artifact and a link artifact that share 200 files into one runtime concurrently, 20 times, and checks both depot sources are byte-for-byte intact. It fails 10/10 on the first revision of this PR and passes on master and on this PR (20/20 under -race).
  • TestLinkContentsConcurrentSameDest links 8 sources providing the same files into one destination concurrently. It fails on master with "file exists" and passes here.
  • TestLinkSkipsExistingTarget checks that an existing target is left intact.
  • TestLinkContentsEquivalence covers regular directories, a symlink to a file and a symlink to a directory. It passes on master's implementation too, so the refactor doesn't change linking behavior.

go test -race ./internal/smartlink/ ./pkg/runtime/ passes on Linux. On Windows the new smartlink tests pass. The existing TestLinkContentsWithCircularLink fails on the benchmark machine both here and on master, because creating its symlink fixture needs symlink privileges ("A required privilege is not held by the client").

Not addressed (follow-ups)

  • Existing depot corruption risk with file transforms. ApplyFileTransforms runs after DeployViaCopy, outside the lock, and writes files in place. If another artifact's link already occupies a transformed path (copy skipped it as existing), the transform writes into that artifact's depot file. This can happen regardless of concurrency, depending on install order. A fix is to have transforms replace the file (write and rename) rather than write in place.
  • Windows Defender exclusions for the depot/install dir: likely a large real-world factor, but it modifies the user's AV configuration, so it's a product decision.
  • The copy path (transforms, no hard link support, portable mode) still writes every file and stays serialized.

🤖 Generated with Claude Code

@mitchell-as

Copy link
Copy Markdown
Contributor

FYI I ran the "Test: critical" integration tests to make sure this isn't breaking anything unexpected. The failures that did occur are sporadic and unrelated to this PR.

Comment thread pkg/runtime/depot.go Outdated
// artifacts can be linked into the runtime concurrently (the install worker pool runs several at once).
// This is safe because linking touches no shared in-memory depot state (d.Exists guards its own access
// via mapMutex, and d.depotPath is immutable after construction), directory creation is idempotent, and
// smartlink treats an already-existing target as a skip. This is a major win on Windows where each file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not terribly comfortable with this assertion. It's probably referring to this code in smartlink.go:

	// Multiple artifacts can supply the same file. We do not have a better solution for this at the moment other than
	// favouring the first one encountered.
	if fileutils.TargetExists(dest) {
		logging.Warning("Skipping linking '%s' to '%s' as it already exists", src, dest)
		return nil
	}

If we remove the fs mutex and then come up with a better solution, we lose the mutex guarantee. If we want to remove the mutex, we should update the referenced code's comment to mention the lack of an existing mutex (the one we're removing).

I like the additional comment above DeployViaCopy that we need the mutex to avoid copying to the same destination, but I'm not convinced we can forgo the need for mutex when creating links.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could try making fsMutex an RWMutex and have DeployViaLink take an RLock and DeployViaCopy/Put/Undeploy take Lock. This way the safer operations which are linking can be done in parallel but copying which isn't as safe still has to be sequential.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right not to trust this. Removing the lock outright was unsafe: a copy deployment checks that a destination is free and then calls os.Create. If a concurrent link put a link to another artifact's depot file there in between, the copy wrote through it and overwrote the depot file. TestConcurrentCopyAndLinkKeepDepotIntact reproduces that in 10/10 runs without the lock.

I've gone with the RWMutex suggestion (365652c): DeployViaLink takes RLock, and DeployViaCopy, Put and Undeploy take Lock. Concurrent links only race with each other, which is safe: creating a link never writes through an existing file, and an "already exists" error from the link call is treated as the existing first-one-wins skip. The shared-file comment in smartlink.go now says that artifacts may be linked concurrently and that the caller must not copy at the same time, so a future change there can see what it relies on.

The read lock costs nothing measurable on Windows (10.2 s vs 10.4 s without any lock); the benchmark table is in the PR description.

@MDrakos MDrakos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR needs some tests, especially to test the concurrency element which this PR addresses.

Some ideas from my robot all run with -race:

  • A deterministic unit test for the new os.IsExist branch: pre-create the target,
    call Link, assert it returns nil and leaves the existing target intact.
  • A concurrent same-file link test in smartlink (two goroutines, same dest) to hit
    the TargetExists -> linkFile -> os.IsExist path.
  • If we take the RWMutex: a pkg/runtime test deploying a copy artifact and a link
    artifact that share a file concurrently, asserting the depot source stays
    byte-for-byte identical — the regression guard for the truncation case.
  • A quick equivalence test for the Link refactor (regular dir / symlink->file /
    symlink->dir), since only the circular case is covered today.

Comment thread pkg/runtime/depot.go Outdated
// artifacts can be linked into the runtime concurrently (the install worker pool runs several at once).
// This is safe because linking touches no shared in-memory depot state (d.Exists guards its own access
// via mapMutex, and d.depotPath is immutable after construction), directory creation is idempotent, and
// smartlink treats an already-existing target as a skip. This is a major win on Windows where each file

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could try making fsMutex an RWMutex and have DeployViaLink take an RLock and DeployViaCopy/Put/Undeploy take Lock. This way the safer operations which are linking can be done in parallel but copying which isn't as safe still has to be sequential.

icanhasmath and others added 4 commits October 20, 2026 23:42
smartlink.Link resolved both src and dest via ResolveUniquePath for every
entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName,
which is a syscall, so installing a runtime with many files performed two
extra syscalls per file - a major contributor to slow Windows installs.

The tree root is already resolved by LinkContents (and now by the public
Link), and descendant paths are built by joining an already-resolved parent
with real (long) entry names, so they need no further resolution. Split the
recursion into an unexported link() worker that assumes resolved inputs and
does not re-resolve, keeping resolution only for the rare symlink-to-file
case where the pre-existing behavior links to the dereferenced target.

Also treat an "already exists" error from linkFile the same as the existing
TargetExists skip, so concurrent deploys of a shared file degrade to the
documented "first one wins" behavior rather than failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DeployViaLink held the depot-wide fsMutex for the entire link operation, so
even though the install worker pool runs several artifacts at once, the
actual file linking was serialized to one artifact at a time. On Windows,
where each link is an individual (relatively slow) syscall, this made the
install phase far slower than it needed to be.

Linking touches no shared in-memory depot state (Exists guards its own map
via mapMutex and depotPath is immutable after construction), directory
creation is idempotent, and smartlink now treats an already-existing target
as a skip, so it is safe to run without the lock. Drop it from DeployViaLink
so linking parallelizes across the worker pool.

DeployViaCopy keeps the lock: two artifacts copying the same file would race
on os.Create and could corrupt the destination. That path is only used for
artifacts needing file transforms, when hard links are unsupported, or in
portable mode, so it remains the less common case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Linking without the depot lock let a concurrent copy deployment write
through a link created moments earlier: copyFiles checks the destination
is free, a link to another artifact's depot file appears there, and
os.Create then truncates and overwrites that depot file, corrupting the
shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in
every run without the lock.

Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links
still run concurrently, while copies, Put and Undeploy hold it for
writing. Also fix the non-constant format string in DeployViaCopy, which
made go test's vet step fail the whole package.

Add smartlink tests for an existing target, concurrent links to the same
destination, and the Link refactor's handling of directories and
symlinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its
parent and TargetExists before linking it, plus another IsDir inside the
Windows linkFile. Each is a file open on Windows. Instead:

- take the entry type from the directory listing (os.ReadDir already
  has it), so only symlinks need a Stat;
- rely on the parent directory existing, since the recursion creates it
  (Link creates it for its single entry);
- drop the TargetExists pre-check and treat the link call's "already
  exists" error as the skip, which already covered concurrent links;
- check for a directory only on the symlink-to-directory path, which is
  the only one that can pass one to linkFile.

A regular file now costs one syscall: the link itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@icanhasmath

Copy link
Copy Markdown
Collaborator Author

@MDrakos thanks for the list. All four tests are added, and the race checks pass under -race:

  • Existing target: TestLinkSkipsExistingTarget pre-creates the target, calls Link, and checks it returns nil and leaves the target intact.
  • Concurrent same-file links: TestLinkContentsConcurrentSameDest links 8 sources that provide the same files into one destination concurrently. It fails on master with "file exists" and passes here.
  • Copy and link sharing a file: TestConcurrentCopyAndLinkKeepDepotIntact deploys a copy artifact and a link artifact that share 200 files into one runtime concurrently, 20 times, and checks both depot sources are byte-for-byte intact. It fails 10/10 on the first revision of this PR and passes 20/20 with the RWMutex.
  • Link refactor equivalence: TestLinkContentsEquivalence covers regular directories, a symlink to a file and a symlink to a directory. It also passes against master's implementation, so the refactor doesn't change linking behavior.

I also benchmarked the change natively on Windows. With a further commit that drops redundant per-file metadata calls, linking a 22k-file runtime went from 75 s to 7 s; details are in the updated description.

@icanhasmath
icanhasmath merged commit 8f473b3 into master Oct 21, 2026
10 of 13 checks passed
@icanhasmath
icanhasmath deleted the windows-install-perf branch October 21, 2026 20:16
@icanhasmath icanhasmath mentioned this pull request Oct 21, 2026
icanhasmath added a commit that referenced this pull request Oct 21, 2026
* Resolve link paths once per tree instead of once per file

smartlink.Link resolved both src and dest via ResolveUniquePath for every
entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName,
which is a syscall, so installing a runtime with many files performed two
extra syscalls per file - a major contributor to slow Windows installs.

The tree root is already resolved by LinkContents (and now by the public
Link), and descendant paths are built by joining an already-resolved parent
with real (long) entry names, so they need no further resolution. Split the
recursion into an unexported link() worker that assumes resolved inputs and
does not re-resolve, keeping resolution only for the rare symlink-to-file
case where the pre-existing behavior links to the dereferenced target.

Also treat an "already exists" error from linkFile the same as the existing
TargetExists skip, so concurrent deploys of a shared file degrade to the
documented "first one wins" behavior rather than failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Deploy linked artifacts concurrently instead of serializing them

DeployViaLink held the depot-wide fsMutex for the entire link operation, so
even though the install worker pool runs several artifacts at once, the
actual file linking was serialized to one artifact at a time. On Windows,
where each link is an individual (relatively slow) syscall, this made the
install phase far slower than it needed to be.

Linking touches no shared in-memory depot state (Exists guards its own map
via mapMutex and depotPath is immutable after construction), directory
creation is idempotent, and smartlink now treats an already-existing target
as a skip, so it is safe to run without the lock. Drop it from DeployViaLink
so linking parallelizes across the worker pool.

DeployViaCopy keeps the lock: two artifacts copying the same file would race
on os.Create and could corrupt the destination. That path is only used for
artifacts needing file transforms, when hard links are unsupported, or in
portable mode, so it remains the less common case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Hold a read lock while linking so copies can't write through links

Linking without the depot lock let a concurrent copy deployment write
through a link created moments earlier: copyFiles checks the destination
is free, a link to another artifact's depot file appears there, and
os.Create then truncates and overwrites that depot file, corrupting the
shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in
every run without the lock.

Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links
still run concurrently, while copies, Put and Undeploy hold it for
writing. Also fix the non-constant format string in DeployViaCopy, which
made go test's vet step fail the whole package.

Add smartlink tests for an existing target, concurrent links to the same
destination, and the Link refactor's handling of directories and
symlinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip per-file metadata calls when linking

link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its
parent and TargetExists before linking it, plus another IsDir inside the
Windows linkFile. Each is a file open on Windows. Instead:

- take the entry type from the directory listing (os.ReadDir already
  has it), so only symlinks need a Stat;
- rely on the parent directory existing, since the recursion creates it
  (Link creates it for its single entry);
- drop the TargetExists pre-check and treat the link call's "already
  exists" error as the skip, which already covered concurrent links;
- check for a directory only on the symlink-to-directory path, which is
  the only one that can pass one to linkFile.

A regular file now costs one syscall: the link itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
icanhasmath added a commit that referenced this pull request Oct 21, 2026
* Resolve link paths once per tree instead of once per file

smartlink.Link resolved both src and dest via ResolveUniquePath for every
entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName,
which is a syscall, so installing a runtime with many files performed two
extra syscalls per file - a major contributor to slow Windows installs.

The tree root is already resolved by LinkContents (and now by the public
Link), and descendant paths are built by joining an already-resolved parent
with real (long) entry names, so they need no further resolution. Split the
recursion into an unexported link() worker that assumes resolved inputs and
does not re-resolve, keeping resolution only for the rare symlink-to-file
case where the pre-existing behavior links to the dereferenced target.

Also treat an "already exists" error from linkFile the same as the existing
TargetExists skip, so concurrent deploys of a shared file degrade to the
documented "first one wins" behavior rather than failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Deploy linked artifacts concurrently instead of serializing them

DeployViaLink held the depot-wide fsMutex for the entire link operation, so
even though the install worker pool runs several artifacts at once, the
actual file linking was serialized to one artifact at a time. On Windows,
where each link is an individual (relatively slow) syscall, this made the
install phase far slower than it needed to be.

Linking touches no shared in-memory depot state (Exists guards its own map
via mapMutex and depotPath is immutable after construction), directory
creation is idempotent, and smartlink now treats an already-existing target
as a skip, so it is safe to run without the lock. Drop it from DeployViaLink
so linking parallelizes across the worker pool.

DeployViaCopy keeps the lock: two artifacts copying the same file would race
on os.Create and could corrupt the destination. That path is only used for
artifacts needing file transforms, when hard links are unsupported, or in
portable mode, so it remains the less common case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Hold a read lock while linking so copies can't write through links

Linking without the depot lock let a concurrent copy deployment write
through a link created moments earlier: copyFiles checks the destination
is free, a link to another artifact's depot file appears there, and
os.Create then truncates and overwrites that depot file, corrupting the
shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in
every run without the lock.

Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links
still run concurrently, while copies, Put and Undeploy hold it for
writing. Also fix the non-constant format string in DeployViaCopy, which
made go test's vet step fail the whole package.

Add smartlink tests for an existing target, concurrent links to the same
destination, and the Link refactor's handling of directories and
symlinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip per-file metadata calls when linking

link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its
parent and TargetExists before linking it, plus another IsDir inside the
Windows linkFile. Each is a file open on Windows. Instead:

- take the entry type from the directory listing (os.ReadDir already
  has it), so only symlinks need a Stat;
- rely on the parent directory existing, since the recursion creates it
  (Link creates it for its single entry);
- drop the TargetExists pre-check and treat the link call's "already
  exists" error as the skip, which already covered concurrent links;
- check for a directory only on the symlink-to-directory path, which is
  the only one that can pass one to linkFile.

A regular file now costs one syscall: the link itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
icanhasmath added a commit that referenced this pull request Oct 23, 2026
* Speed up file installation on Windows (#3832)

* Resolve link paths once per tree instead of once per file

smartlink.Link resolved both src and dest via ResolveUniquePath for every
entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName,
which is a syscall, so installing a runtime with many files performed two
extra syscalls per file - a major contributor to slow Windows installs.

The tree root is already resolved by LinkContents (and now by the public
Link), and descendant paths are built by joining an already-resolved parent
with real (long) entry names, so they need no further resolution. Split the
recursion into an unexported link() worker that assumes resolved inputs and
does not re-resolve, keeping resolution only for the rare symlink-to-file
case where the pre-existing behavior links to the dereferenced target.

Also treat an "already exists" error from linkFile the same as the existing
TargetExists skip, so concurrent deploys of a shared file degrade to the
documented "first one wins" behavior rather than failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Deploy linked artifacts concurrently instead of serializing them

DeployViaLink held the depot-wide fsMutex for the entire link operation, so
even though the install worker pool runs several artifacts at once, the
actual file linking was serialized to one artifact at a time. On Windows,
where each link is an individual (relatively slow) syscall, this made the
install phase far slower than it needed to be.

Linking touches no shared in-memory depot state (Exists guards its own map
via mapMutex and depotPath is immutable after construction), directory
creation is idempotent, and smartlink now treats an already-existing target
as a skip, so it is safe to run without the lock. Drop it from DeployViaLink
so linking parallelizes across the worker pool.

DeployViaCopy keeps the lock: two artifacts copying the same file would race
on os.Create and could corrupt the destination. That path is only used for
artifacts needing file transforms, when hard links are unsupported, or in
portable mode, so it remains the less common case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Hold a read lock while linking so copies can't write through links

Linking without the depot lock let a concurrent copy deployment write
through a link created moments earlier: copyFiles checks the destination
is free, a link to another artifact's depot file appears there, and
os.Create then truncates and overwrites that depot file, corrupting the
shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in
every run without the lock.

Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links
still run concurrently, while copies, Put and Undeploy hold it for
writing. Also fix the non-constant format string in DeployViaCopy, which
made go test's vet step fail the whole package.

Add smartlink tests for an existing target, concurrent links to the same
destination, and the Link refactor's handling of directories and
symlinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip per-file metadata calls when linking

link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its
parent and TargetExists before linking it, plus another IsDir inside the
Windows linkFile. Each is a file open on Windows. Instead:

- take the entry type from the directory listing (os.ReadDir already
  has it), so only symlinks need a Stat;
- rely on the parent directory existing, since the recursion creates it
  (Link creates it for its single entry);
- drop the TargetExists pre-check and treat the link call's "already
  exists" error as the skip, which already covered concurrent links;
- check for a directory only on the symlink-to-directory path, which is
  the only one that can pass one to linkFile.

A regular file now costs one syscall: the link itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8f473b3)

* Remediate CVE-2026-63209 in klauspost/compress (#3858)

Bump github.com/klauspost/compress (indirect, via mholt/archives) from
v1.18.0 to v1.18.7, the first patched release for GHSA-259r-337f-4rfw
(integer overflow and out-of-bounds write in s2). Re-vendored.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b09f7ee)

* Support dependencies for encrypted private ingredients (#3857)

* Support dependencies for encrypted private ingredients

`state publish --build` can now declare and edit the dependencies of an
encrypted private ingredient:

- Read [project].dependencies from pyproject.toml and publish them as
  runtime dependencies on language/python packages, with names normalized
  by the inventory API. Version ranges carry over (~= and ==X.* are
  expanded). OS and Python version markers become conditions
  (kernel/Windows|Linux|Darwin, language/python); unsupported markers are
  skipped with a notice. The wheel's METADATA records them as
  Requires-Dist.
- Read the --meta file before building, so a meta file's name, version
  and description feed the wheel (flags > meta file > pyproject.toml).
- Add `state export ingredient-meta <namespace>/<name>[@Version]`, which
  writes an existing ingredient's meta file for use with --edit --meta.

Fix --edit along the way: keep the meta file's dependencies and authors
instead of overwriting them with the previous revision's, keep inherited
dependency types and conditions, add the private-builder dependency only
once, and reject author/description changes on an existing version up
front instead of failing in the API. InventoryRequirementsToString no
longer panics on requirements without a version.

Add a readme walking through the publish and edit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review: keep inherited deps, edit the meta file's version

- When every pyproject.toml dependency is skipped, keep the dependencies
  already in place (on --edit, those inherited from the previous
  revision) instead of replacing them with an empty list.
- Let a meta file's version select the version to publish or edit, like
  --version (which still takes precedence). An exported meta file passed
  back with `state publish --edit --meta` without --build was editing
  the latest version instead of the exported one.

Add tests for both, for the up-front author/description rejection on
--edit, for InventoryRequirementsToString with requirements missing a
version or comparator, and for inherited conditions that carry
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 44e750f)

* Add 0.48.2 changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@icanhasmath icanhasmath mentioned this pull request Oct 23, 2026
MDrakos pushed a commit that referenced this pull request Oct 23, 2026
* Update version.txt

* 0.48.2-RC1: #3832, #3858, #3857 on top of 0.48.1-RC4 (#3863)

* Speed up file installation on Windows (#3832)

* Resolve link paths once per tree instead of once per file

smartlink.Link resolved both src and dest via ResolveUniquePath for every
entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName,
which is a syscall, so installing a runtime with many files performed two
extra syscalls per file - a major contributor to slow Windows installs.

The tree root is already resolved by LinkContents (and now by the public
Link), and descendant paths are built by joining an already-resolved parent
with real (long) entry names, so they need no further resolution. Split the
recursion into an unexported link() worker that assumes resolved inputs and
does not re-resolve, keeping resolution only for the rare symlink-to-file
case where the pre-existing behavior links to the dereferenced target.

Also treat an "already exists" error from linkFile the same as the existing
TargetExists skip, so concurrent deploys of a shared file degrade to the
documented "first one wins" behavior rather than failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Deploy linked artifacts concurrently instead of serializing them

DeployViaLink held the depot-wide fsMutex for the entire link operation, so
even though the install worker pool runs several artifacts at once, the
actual file linking was serialized to one artifact at a time. On Windows,
where each link is an individual (relatively slow) syscall, this made the
install phase far slower than it needed to be.

Linking touches no shared in-memory depot state (Exists guards its own map
via mapMutex and depotPath is immutable after construction), directory
creation is idempotent, and smartlink now treats an already-existing target
as a skip, so it is safe to run without the lock. Drop it from DeployViaLink
so linking parallelizes across the worker pool.

DeployViaCopy keeps the lock: two artifacts copying the same file would race
on os.Create and could corrupt the destination. That path is only used for
artifacts needing file transforms, when hard links are unsupported, or in
portable mode, so it remains the less common case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Hold a read lock while linking so copies can't write through links

Linking without the depot lock let a concurrent copy deployment write
through a link created moments earlier: copyFiles checks the destination
is free, a link to another artifact's depot file appears there, and
os.Create then truncates and overwrites that depot file, corrupting the
shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in
every run without the lock.

Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links
still run concurrently, while copies, Put and Undeploy hold it for
writing. Also fix the non-constant format string in DeployViaCopy, which
made go test's vet step fail the whole package.

Add smartlink tests for an existing target, concurrent links to the same
destination, and the Link refactor's handling of directories and
symlinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip per-file metadata calls when linking

link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its
parent and TargetExists before linking it, plus another IsDir inside the
Windows linkFile. Each is a file open on Windows. Instead:

- take the entry type from the directory listing (os.ReadDir already
  has it), so only symlinks need a Stat;
- rely on the parent directory existing, since the recursion creates it
  (Link creates it for its single entry);
- drop the TargetExists pre-check and treat the link call's "already
  exists" error as the skip, which already covered concurrent links;
- check for a directory only on the symlink-to-directory path, which is
  the only one that can pass one to linkFile.

A regular file now costs one syscall: the link itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8f473b3)

* Remediate CVE-2026-63209 in klauspost/compress (#3858)

Bump github.com/klauspost/compress (indirect, via mholt/archives) from
v1.18.0 to v1.18.7, the first patched release for GHSA-259r-337f-4rfw
(integer overflow and out-of-bounds write in s2). Re-vendored.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b09f7ee)

* Support dependencies for encrypted private ingredients (#3857)

* Support dependencies for encrypted private ingredients

`state publish --build` can now declare and edit the dependencies of an
encrypted private ingredient:

- Read [project].dependencies from pyproject.toml and publish them as
  runtime dependencies on language/python packages, with names normalized
  by the inventory API. Version ranges carry over (~= and ==X.* are
  expanded). OS and Python version markers become conditions
  (kernel/Windows|Linux|Darwin, language/python); unsupported markers are
  skipped with a notice. The wheel's METADATA records them as
  Requires-Dist.
- Read the --meta file before building, so a meta file's name, version
  and description feed the wheel (flags > meta file > pyproject.toml).
- Add `state export ingredient-meta <namespace>/<name>[@Version]`, which
  writes an existing ingredient's meta file for use with --edit --meta.

Fix --edit along the way: keep the meta file's dependencies and authors
instead of overwriting them with the previous revision's, keep inherited
dependency types and conditions, add the private-builder dependency only
once, and reject author/description changes on an existing version up
front instead of failing in the API. InventoryRequirementsToString no
longer panics on requirements without a version.

Add a readme walking through the publish and edit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review: keep inherited deps, edit the meta file's version

- When every pyproject.toml dependency is skipped, keep the dependencies
  already in place (on --edit, those inherited from the previous
  revision) instead of replacing them with an empty list.
- Let a meta file's version select the version to publish or edit, like
  --version (which still takes precedence). An exported meta file passed
  back with `state publish --edit --meta` without --build was editing
  the latest version instead of the exported one.

Add tests for both, for the up-front author/description rejection on
--edit, for InventoryRequirementsToString with requirements missing a
version or comparator, and for inherited conditions that carry
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 44e750f)

* Add 0.48.2 changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* check-format: don't fail when a PR changes no files.

With an empty diff, grep -v exits 1 and set -e aborts the script with no
output, so a PR that changes nothing (e.g. merging beta into a version
branch) fails Check Format on every OS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update to Go 1.26.10 and golang.org/x/net v0.60.0 to remediate CVEs.

The CI Trivy scan fails on 13 Go stdlib CVEs (fixed in 1.26.9/1.26.10)
and 11 golang.org/x/net findings (fixed in v0.60.0) published since the
0.48.1 scan. Re-vendored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Compare RSA keys by value in keypair tests.

The tests compared a generated keypair with the same key parsed back from
PEM using deep struct equality, which also compares rsa.PrivateKey's
precomputed internals. Under Go 1.26.10 those can differ (e.g. a leading
zero byte in dP) even though the keys are identical, so the tests failed
intermittently. Compare with rsa.PrivateKey.Equal instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Allow for slow sw_vers in the macOS OSVersion performance test. (#3866)

TestPerformance/OSVersion averages ten calls against a 10ms budget, so
the single uncached call bounds it at ~100ms. On macOS that call shells
out to sw_vers, which took 115-716ms on macos-latest runners and failed
3 of 4 runs. Cached calls take nanoseconds. Keep 10ms elsewhere; allow
100ms on darwin.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: mitchell <mitchellb@activestate.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
icanhasmath added a commit that referenced this pull request Nov 2, 2026
* 0.48.2-RC1 (#3864)

* Update version.txt

* 0.48.2-RC1: #3832, #3858, #3857 on top of 0.48.1-RC4 (#3863)

* Speed up file installation on Windows (#3832)

* Resolve link paths once per tree instead of once per file

smartlink.Link resolved both src and dest via ResolveUniquePath for every
entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName,
which is a syscall, so installing a runtime with many files performed two
extra syscalls per file - a major contributor to slow Windows installs.

The tree root is already resolved by LinkContents (and now by the public
Link), and descendant paths are built by joining an already-resolved parent
with real (long) entry names, so they need no further resolution. Split the
recursion into an unexported link() worker that assumes resolved inputs and
does not re-resolve, keeping resolution only for the rare symlink-to-file
case where the pre-existing behavior links to the dereferenced target.

Also treat an "already exists" error from linkFile the same as the existing
TargetExists skip, so concurrent deploys of a shared file degrade to the
documented "first one wins" behavior rather than failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Deploy linked artifacts concurrently instead of serializing them

DeployViaLink held the depot-wide fsMutex for the entire link operation, so
even though the install worker pool runs several artifacts at once, the
actual file linking was serialized to one artifact at a time. On Windows,
where each link is an individual (relatively slow) syscall, this made the
install phase far slower than it needed to be.

Linking touches no shared in-memory depot state (Exists guards its own map
via mapMutex and depotPath is immutable after construction), directory
creation is idempotent, and smartlink now treats an already-existing target
as a skip, so it is safe to run without the lock. Drop it from DeployViaLink
so linking parallelizes across the worker pool.

DeployViaCopy keeps the lock: two artifacts copying the same file would race
on os.Create and could corrupt the destination. That path is only used for
artifacts needing file transforms, when hard links are unsupported, or in
portable mode, so it remains the less common case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Hold a read lock while linking so copies can't write through links

Linking without the depot lock let a concurrent copy deployment write
through a link created moments earlier: copyFiles checks the destination
is free, a link to another artifact's depot file appears there, and
os.Create then truncates and overwrites that depot file, corrupting the
shared cache. TestConcurrentCopyAndLinkKeepDepotIntact reproduces it in
every run without the lock.

Make fsMutex an RWMutex: DeployViaLink holds it for reading, so links
still run concurrently, while copies, Put and Undeploy hold it for
writing. Also fix the non-constant format string in DeployViaCopy, which
made go test's vet step fail the whole package.

Add smartlink tests for an existing target, concurrent links to the same
destination, and the Link refactor's handling of directories and
symlinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip per-file metadata calls when linking

link() checked each file with IsDir, IsSymlink, MkdirUnlessExists on its
parent and TargetExists before linking it, plus another IsDir inside the
Windows linkFile. Each is a file open on Windows. Instead:

- take the entry type from the directory listing (os.ReadDir already
  has it), so only symlinks need a Stat;
- rely on the parent directory existing, since the recursion creates it
  (Link creates it for its single entry);
- drop the TargetExists pre-check and treat the link call's "already
  exists" error as the skip, which already covered concurrent links;
- check for a directory only on the symlink-to-directory path, which is
  the only one that can pass one to linkFile.

A regular file now costs one syscall: the link itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8f473b3)

* Remediate CVE-2026-63209 in klauspost/compress (#3858)

Bump github.com/klauspost/compress (indirect, via mholt/archives) from
v1.18.0 to v1.18.7, the first patched release for GHSA-259r-337f-4rfw
(integer overflow and out-of-bounds write in s2). Re-vendored.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b09f7ee)

* Support dependencies for encrypted private ingredients (#3857)

* Support dependencies for encrypted private ingredients

`state publish --build` can now declare and edit the dependencies of an
encrypted private ingredient:

- Read [project].dependencies from pyproject.toml and publish them as
  runtime dependencies on language/python packages, with names normalized
  by the inventory API. Version ranges carry over (~= and ==X.* are
  expanded). OS and Python version markers become conditions
  (kernel/Windows|Linux|Darwin, language/python); unsupported markers are
  skipped with a notice. The wheel's METADATA records them as
  Requires-Dist.
- Read the --meta file before building, so a meta file's name, version
  and description feed the wheel (flags > meta file > pyproject.toml).
- Add `state export ingredient-meta <namespace>/<name>[@Version]`, which
  writes an existing ingredient's meta file for use with --edit --meta.

Fix --edit along the way: keep the meta file's dependencies and authors
instead of overwriting them with the previous revision's, keep inherited
dependency types and conditions, add the private-builder dependency only
once, and reject author/description changes on an existing version up
front instead of failing in the API. InventoryRequirementsToString no
longer panics on requirements without a version.

Add a readme walking through the publish and edit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review: keep inherited deps, edit the meta file's version

- When every pyproject.toml dependency is skipped, keep the dependencies
  already in place (on --edit, those inherited from the previous
  revision) instead of replacing them with an empty list.
- Let a meta file's version select the version to publish or edit, like
  --version (which still takes precedence). An exported meta file passed
  back with `state publish --edit --meta` without --build was editing
  the latest version instead of the exported one.

Add tests for both, for the up-front author/description rejection on
--edit, for InventoryRequirementsToString with requirements missing a
version or comparator, and for inherited conditions that carry
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 44e750f)

* Add 0.48.2 changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* check-format: don't fail when a PR changes no files.

With an empty diff, grep -v exits 1 and set -e aborts the script with no
output, so a PR that changes nothing (e.g. merging beta into a version
branch) fails Check Format on every OS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update to Go 1.26.10 and golang.org/x/net v0.60.0 to remediate CVEs.

The CI Trivy scan fails on 13 Go stdlib CVEs (fixed in 1.26.9/1.26.10)
and 11 golang.org/x/net findings (fixed in v0.60.0) published since the
0.48.1 scan. Re-vendored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Compare RSA keys by value in keypair tests.

The tests compared a generated keypair with the same key parsed back from
PEM using deep struct equality, which also compares rsa.PrivateKey's
precomputed internals. Under Go 1.26.10 those can differ (e.g. a leading
zero byte in dP) even though the keys are identical, so the tests failed
intermittently. Compare with rsa.PrivateKey.Equal instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Allow for slow sw_vers in the macOS OSVersion performance test. (#3866)

TestPerformance/OSVersion averages ten calls against a 10ms budget, so
the single uncached call bounds it at ~100ms. On macOS that call shells
out to sw_vers, which took 115-716ms on macos-latest runners and failed
3 of 4 runs. Cached calls take nanoseconds. Keep 10ms elsewhere; allow
100ms on darwin.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: mitchell <mitchellb@activestate.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* 0.48.2-RC2 (#3870)

* Update version.txt

* Fix state publish meta data issues found testing 0.48.2-RC1 (#3867)

* Fix state publish meta data issues found testing 0.48.2-RC1

- state export ingredient-meta writes plain yaml, unwrapped and without
  field help comments, so the file can be passed back with --meta.
- Dependencies and conditions without a version range (a bare name in
  pyproject.toml, --depend without a version, or a condition inherited
  without requirements) get >=0. The platform rejected them with
  "Expecting an OBJECT with a field named 'versionRequirements'".
- When --build can't determine the name or version, say which is
  missing, why pyproject.toml didn't provide it (no file, not in
  [project], or listed in [project].dynamic), and how to set it.
- Answering 'n' at the publish confirmation offers to edit the meta data
  in the editor or save it to <name>-<version>.meta.yaml with the command
  to publish it, instead of only cancelling. With --build or --edit, a
  name, namespace or version change in the editor is saved to a file
  instead, as the package was already built and looked up with them.
- Fields removed in the editor are removed from the request, and the
  publish API error is shown instead of <no value>.

The platform requires the source with every publish, so --edit still
needs --build or an archive; the readme now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update echo to v4.16.1 and drop an unused publish error argument

echo v4.15.4 has GHSA-cxf3-9h37-7rr8 (critical), GHSA-9969-6676-49h5,
GHSA-f48p-c9v4-h7xv and GHSA-h7wr-284f-5qvc, which failed the Scan job.

err_uploadingredient_publish is defined in en-us.yaml without a {{.V0}},
so the joined message passed to it was never shown. The wrapped error is
already printed beneath it, so drop the argument and the changelog claim.

* Keep the API error in the publish failure message

err_uploadingredient_publish in en-us.yaml ends in "Error received: {{.V0}}"
and overrides the fallback text, so without the argument the message
reads "<no value>". The wrapped API error isn't user-facing, so it isn't
printed beneath it either; it only reaches the log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix TestOpen failing on Windows; correct the publish error comment

TestOpen removed its temp file while the hail watcher could still be
reading it for a later write event: the remove was deferred after the
cancel, so it ran first. On Windows an open file can't be removed. Stop
the watcher and drain its channel before removing the file.

The publish error comment said the wrapped API error is never printed;
a transport failure becomes a user-facing BuildPlannerError and is. Only
an error reported in the publish result (an errs.New) relies on {{.V0}}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Validate each confirm pass and test the publish fixes through their call paths

Address review on #3867:

- Run the namespace and checkEditKeepsIdentity checks at the top of each
  pass of the confirm loop, now confirmRequest, so a namespace cleared or
  a description added in the editor is rejected with a clear message
  instead of failing in MarshalJSON or the API.
- Add tests that fail if a fix is reverted: confirmRequest with a fake
  prompter (>=0 filling, editor removing fields, cleared namespace,
  description on a revision, private-builder restored, save, cancel),
  using the test binary as $EDITOR so it also runs on Windows;
  publishFailedError keeping the API error; metaFromIngredient filling
  conditions; and writeIngredientMeta writing plain yaml that
  round-trips.

FetchMeta's pure part moves into metaFromIngredient and the export's
writing into writeIngredientMeta so they can be tested without the API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Fix CI for 0.48.2-RC2: x/net CVEs, publish --editor, and runner-specific test failures (#3869)

* Update golang.org/x/net to v0.62.0 for CVE-2026-78661 and CVE-2026-97045

The Scan job failed on both: x/net v0.60.0 is affected and v0.62.0 fixes them.
go get also raised x/crypto, x/sys, x/term, x/text, x/sync and x/tools to the
versions x/net v0.62.0 requires.

* Don't search every namespace when publishing a new ingredient without one

state publish --editor outside a project has no namespace until the editor
fills it in, so Run searched every namespace for the name derived from the file.
Logged in, that search took long enough to time out ("Could not search for
ingredient"), so the editor never opened. This is why the "New ingredient with
editor flag" integration test has failed nightly on master since at least
September. Its first match could also be any ingredient with that name.

Skip the lookup for a new ingredient without a namespace; confirmRequest
requires one and checkNamespaceEdit checks the version once it is filled in.
--edit still searches, since it needs to find the ingredient.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Let state publish --editor continue when no editor can be opened

Without EDITOR set and without xdg-open, as on many headless Linux machines
and the ubuntu-24.04-arm runner, OpenEditor fails and publish exited with
"Could not publish ingredient. Error received: <no value>", even though the
notice just before it says the file can be edited by hand. Say that no editor
could be opened and wait for the confirmation as usual instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix integration tests that fail only because of the runner or branch name

- TestPublishBuildEncrypted installs into a Python project that state init
  creates without a Linux/arm64 platform, so skip it there, like the other
  platform-specific skips. Integration tests only run on ARM for labelled PRs.
- PerformanceIntegrationTestSuite's TearDownSuite stopped state-svc even when
  its tests were skipped by tag. On macOS a long branch channel makes the socket
  path too long to dial, so every labelled PR from such a branch failed there.
  Only stop the service when the Performance tests ran, using a new
  tagsuite.RunsForTags that OnlyRunForTags is now built on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Don't open a real editor in the publish --editor integration test

The test writes its edits to the file itself, but with no EDITOR set, macOS
`open` hands the file to whatever app the runner has registered. On
macos-latest, publish then read the file back without the test's edits and
failed with "You have to supply the namespace". Point EDITOR at a command that
doesn't exist, so publish takes the "edit the file by hand" path on every OS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the publish --editor integration test's flaky file path match

editorFileRx matched "file:\s*?(.*?)\.\s", whose lazy \s*? left the space
after "file:" in the captured path whenever the temp file name had 9 digits
instead of 10 (about one run in four). The test then wrote its edits to
" /tmp/NNN.ingredient.yaml", a different file, so publish read the original
back and failed with "You have to supply the namespace". The last commit's
macOS explanation was wrong; this was the cause on every OS.

Reproduced by driving a dev build through termtest 12 times: three runs captured
the leading space before this change and none after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* Make the state publish meta data clearer to review and edit (#3868)

* Update golang.org/x/net to v0.62.0 for CVE-2026-78661 and CVE-2026-97045

The Scan job failed on both: x/net v0.60.0 is affected and v0.62.0 fixes them.
go get also raised x/crypto, x/sys, x/term, x/text, x/sync and x/tools to the
versions x/net v0.62.0 requires.

* Don't search every namespace when publishing a new ingredient without one

state publish --editor outside a project has no namespace until the editor
fills it in, so Run searched every namespace for the name derived from the file.
Logged in, that search took long enough to time out ("Could not search for
ingredient"), so the editor never opened. This is why the "New ingredient with
editor flag" integration test has failed nightly on master since at least
September. Its first match could also be any ingredient with that name.

Skip the lookup for a new ingredient without a namespace; confirmRequest
requires one and checkNamespaceEdit checks the version once it is filled in.
--edit still searches, since it needs to find the ingredient.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Let state publish --editor continue when no editor can be opened

Without EDITOR set and without xdg-open, as on many headless Linux machines
and the ubuntu-24.04-arm runner, OpenEditor fails and publish exited with
"Could not publish ingredient. Error received: <no value>", even though the
notice just before it says the file can be edited by hand. Say that no editor
could be opened and wait for the confirmation as usual instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix integration tests that fail only because of the runner or branch name

- TestPublishBuildEncrypted installs into a Python project that state init
  creates without a Linux/arm64 platform, so skip it there, like the other
  platform-specific skips. Integration tests only run on ARM for labelled PRs.
- PerformanceIntegrationTestSuite's TearDownSuite stopped state-svc even when
  its tests were skipped by tag. On macOS a long branch channel makes the socket
  path too long to dial, so every labelled PR from such a branch failed there.
  Only stop the service when the Performance tests ran, using a new
  tagsuite.RunsForTags that OnlyRunForTags is now built on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Don't open a real editor in the publish --editor integration test

The test writes its edits to the file itself, but with no EDITOR set, macOS
`open` hands the file to whatever app the runner has registered. On
macos-latest, publish then read the file back without the test's edits and
failed with "You have to supply the namespace". Point EDITOR at a command that
doesn't exist, so publish takes the "edit the file by hand" path on every OS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the publish --editor integration test's flaky file path match

editorFileRx matched "file:\s*?(.*?)\.\s", whose lazy \s*? left the space
after "file:" in the captured path whenever the temp file name had 9 digits
instead of 10 (about one run in four). The test then wrote its edits to
" /tmp/NNN.ingredient.yaml", a different file, so publish read the original
back and failed with "You have to supply the namespace". The last commit's
macOS explanation was wrong; this was the cause on every OS.

Reproduced by driving a dev build through termtest 12 times: three runs captured
the leading space before this change and none after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Make the state publish meta data clearer to review and edit

Follow-up to #3867 from testing the 0.48.2-RC2 build:

- Without --namespace, a --build publish from a pyproject.toml now defaults
  to private/<org>/python (otherwise private/<org>, as before).
- A new ingredient's namespace can be changed in the editor. It is checked
  like --namespace and put back, with the reason, if it can't be used.
- The confirmation and editor say up front which fields can't be changed and
  why, and the error for changing one names it.
- The preview and editor show plain yaml without a help comment on every
  field; the fields are described in the publish guide instead. The
  yaml-comment module is no longer used and is removed.
- state export ingredient-meta now includes the ingredient's description.
  state publish --edit no longer refuses a different description on an
  existing version: it keeps the current one and says the change is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: mitchell <mitchellb@activestate.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants