fix(ssh): persist sandbox host identities

[quocanh261997]

Summary

Sandbox SSH host keys currently change when the supervisor restarts, so clients cannot reliably recognize the same sandbox. Give each sandbox ID one Ed25519 host key and expose its public SHA256 fingerprint. Keep the private key in the configured gateway credential store and deliver it only to the supervisor.

Related Issue

Closes #3835

Replaces #4027, which the vouch check closed before contributor approval. The contributor is now vouched.

Changes

• Route failed SSH-key preparation and rejected sandbox creation through the shared cleanup path. Remove the sandbox and key when cleanup succeeds; retain a durable Deleting record when cleanup needs a retry. Keep cleanup running if the caller disconnects, including while waiting for the cleanup lock.
• Persist a sandbox-owned credential handle and fingerprint. Keep identity across stop/start, automatic restarts, and gateway recovery; remove credentials during deletion. Retain cleanup ownership when credential deletion fails and finish staging if a request is interrupted.
• Require the managed supervisor to use its assigned host key. Reject missing, invalid, or mismatched key material instead of replacing an established identity.
• Return the fingerprint on Sandbox and SSH-session responses, sandbox JSON output, and Rust, Python, Go, and TypeScript sandbox references.
• Document identity lifetime, authenticated fingerprint lookup, workload isolation, and matching runtime releases. CLI/TUI automatic verification remains the optional follow-on from the issue.

Testing

• mise run pre-commit passes.
• Unit tests added/updated.
• E2E tests added/updated.
• mise run ci — passed, including workspace and gateway Rust tests, Python tests, Go checks, TypeScript tests, formatting, lint, and compile checks.
• mise run sdk:ts:ci — passed, including 142 tests, coverage, and build validation.
• mise run go:ci — passed.
• Feature-enabled lint of ssh_host_identity — passed.
• Docker CLI conformance — all six scenarios passed.
• Docker SSH identity E2E — one test passed, with no failures or skips. Verified Get/List against the presented SSH fingerprint, a pinned OpenSSH connection after stop/start and managed gateway restart, workload read restrictions, and a new identity after delete/recreate.

Rust checks used LIBRARY_PATH=/opt/homebrew/lib for Homebrew Z3 on macOS. Installed e2fsprogs for the existing VM filesystem tests.

The stock Docker E2E launcher failed locally because its container-side 127.0.0.1 endpoint cannot reach the macOS gateway. The successful conformance run uses an ignored copy of the launcher with host.docker.internal as the supervisor endpoint and the temporary mTLS gateway bound on all interfaces. This local launcher adjustment is excluded from the PR.

Six additional failed-create tests cover a failed credential save and name reuse, failed key deletion after each driver rejection, failed fingerprint publication, failed backend cleanup, and caller cancellation during deletion or while waiting for the cleanup lock.

Added identity tests cover credential-store reconstruction, concurrent candidates, interrupted preparation, deletion retries, failed candidate cleanup, inconsistent published identity, secret redaction, older bootstrap bundles, client conversion, fingerprint responses, and real SSH handshakes with direct and relayed streams after listener restart. A live sandbox test checks Get/List fingerprints, a remembered OpenSSH host key through stop/start and gateway restart, workload access restrictions, and recreation under the same name.

The live test uses the Docker driver. Kubernetes pod rescheduling and other compute-driver lanes have not been exercised locally.

Checklist

• Follows Conventional Commits.
• Commits are signed off (DCO).
• Architecture docs updated where applicable; public behavior and related operating skills are documented.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[quocanh261997]

Local verification evidence

The saved local run passed all six Docker CLI conformance scenarios and the SSH identity E2E test. This report covers the working-tree implementation subsequently committed as e5de0a892813408bf08a36d93ab33b66a9929d18, the PR’s initial implementation commit. Updated verification for the cleanup fix is recorded in this later comment.

Environment: macOS on ARM64, Docker Desktop. Live test run: October 1, 2026. The excerpts below come from the saved test output.

What the live SSH test verified

Check	Result
Sandbox Get and List expose the same public fingerprint	Passed
ssh-keygen reports the same fingerprint for the key actually presented over SSH	Passed
The workload cannot read /.openshell/supervisor/auth.json	Passed
Stop/start preserves the fingerprint and an OpenSSH connection with StrictHostKeyChecking=yes succeeds using the previously saved host key	Passed
Managed gateway restart preserves the fingerprint and the same pinned SSH connection succeeds	Passed
Delete/recreate under the same name produces a different fingerprint	Passed

The test source at this commit contains these assertions. The gateway restart branch ran; its skip message is absent from the saved output.

Actual SSH test output:

running 1 test
test ssh_host_identity_survives_restarts_and_changes_after_recreation ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.25s

Supporting checks

• Docker CLI conformance: smoke, sandbox-lifecycle, file-transfer, mechanistic-proposal, new-hostname-proposal, and policy-local each returned "passed": true; the overall report also returned "passed": true.
• mise run ci completed successfully. Its gateway test summary was 1882 passed; 0 failed; 8 ignored. The seven new SSH credential tests below all passed.
• mise run pre-commit completed successfully before publishing the PR.

Actual SSH credential test output

test ssh_identity::tests::cancelled_prepare_finishes_before_deletion_and_cannot_recreate_identity ... ok
test ssh_identity::tests::concurrent_candidates_keep_one_resolvable_identity ... ok
test ssh_identity::tests::deletion_retries_before_releasing_key_ownership ... ok
test ssh_identity::tests::identity_survives_reload_and_is_never_public ... ok
test ssh_identity::tests::losing_candidate_cleanup_failure_remains_owned_for_deletion ... ok
test ssh_identity::tests::default_credential_store_persists_key_across_runtime_reconstruction ... ok
test ssh_identity::tests::published_identity_is_never_replaced_when_storage_is_inconsistent ... ok

Credential test source

Reproduction and scope

The standard focused Docker command is:

OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity mise run e2e:docker

On this Mac, the stock launcher's container-side 127.0.0.1 endpoint could not reach the host gateway. The successful run used an ignored copy of e2e/with-docker-gateway.sh, with just this Darwin-specific addition after the existing container-network check:

if [ "$(uname -s)" = "Darwin" ]; then
  GATEWAY_BIND_IP="0.0.0.0"
  SUPERVISOR_GATEWAY_HOST="host.docker.internal"
fi

The gateway still used mTLS. The successful local invocation was:

LIBRARY_PATH=/opt/homebrew/lib \
OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity \
OPENSHELL_CONFORMANCE_BIN="$PWD/target/debug/openshell-conformance" \
mise exec -- bash plans/3835-with-docker-gateway-macos.sh \
  bash e2e/rust/e2e-docker.sh __openshell_run_docker_e2e

The launcher adjustment is excluded from the PR. This evidence covers local Docker verification; Kubernetes pod rescheduling and other driver E2E lanes were not exercised. GitHub runner validation is still pending.

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997 for documenting the current-head Docker and SSH verification. I checked the full patch and the creation/cleanup paths those tests exercise; one PR-owned failure path can leave a sandbox record and its host-key credential behind after create reports an error.

Action required: make every post-commit identity failure either fully remove the sandbox and key or durably leave the sandbox in a retryable deletion state.

Blocking findings:

• GATOR-e5de0a89-01: see the inline Warning on the post-commit identity preparation call.

Carried findings:

• None

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: sandbox identity behavior is documented under docs/; related operating skills are updated
• Checks: DCO and vouch gates are green; required branch workflows have not been dispatched for this head
• E2E: test:e2e is required for sandbox lifecycle and credential-flow changes, but dispatch waits until blocking review feedback is resolved
• Head SHA: e5de0a892813408bf08a36d93ab33b66a9929d18
• Base SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2
• Merge base SHA: 021400be8af471f8669369e679de3e18cf0bd672
• Patch ID: accf65c51eed8be52b1848efb2eb9bfbfe089f99
• Gator payload: 10
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[quocanh261997]

🏗️ build-from-issue-agent

Local verification after the failed-create fix

Tested the working tree now committed as a645d9fd5d61993ed8c8f3d848955e38f0e81be5 on October 2, 2026, using macOS ARM64 and Docker Desktop. This covers the fix for the failed-create cleanup finding.

The six new regression tests all passed. They inject failures to check that:

• A failed SSH credential save leaves no sandbox or key, and the name can be reused.
• If key deletion fails after the driver rejects creation (AlreadyExists, FailedPrecondition, or another error), the durable sandbox stays Deleting. Reconciliation then removes the row and key.
• Failed fingerprint publication removes the staged key and sandbox.
• A backend cleanup error does not restore the failed sandbox to Provisioning.
• Cleanup finishes after the caller is canceled, both during driver deletion and while waiting for its lock.

test compute::tests::ssh_credential_delete_failure_keeps_failed_create_deleting_until_reconciliation ... ok
test compute::tests::ssh_credential_store_failure_compensates_create_and_releases_name ... ok
test compute::tests::ssh_failed_create_backend_cleanup_error_does_not_restore_provisioning ... ok
test compute::tests::ssh_failed_create_compensation_survives_cancellation_while_waiting_for_lock ... ok
test compute::tests::ssh_failed_create_compensation_survives_request_cancellation ... ok
test compute::tests::ssh_fingerprint_persistence_failure_compensates_staged_identity ... ok

Full LIBRARY_PATH=/opt/homebrew/lib mise run ci passed, including workspace Rust tests, gateway tests (1888 passed; 0 failed; 8 ignored), Python (258 passed), TypeScript (142 passed), Go checks, and formatting/lint/compile checks. mise run pre-commit passed. The eight ignored gateway tests are existing suite exclusions; all six added regressions ran.

E2E Test Attestation

Gateway mode: Docker. Every live test/scenario executed by the focused launcher passed:

Test / scenario	Result
CLI conformance smoke	Passed
CLI conformance sandbox-lifecycle	Passed
CLI conformance file-transfer	Passed
CLI conformance mechanistic-proposal	Passed
CLI conformance new-hostname-proposal	Passed
CLI conformance policy-local	Passed
ssh_host_identity_survives_restarts_and_changes_after_recreation	Passed

test ssh_host_identity_survives_restarts_and_changes_after_recreation ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.33s

The SSH test checks the published fingerprint against the key actually presented, workload read restrictions, a pinned OpenSSH connection after stop/start and a managed gateway restart, and a different key after delete/recreate. The gateway restart branch ran; no SSH test was skipped.

Command used:

LIBRARY_PATH=/opt/homebrew/lib \
OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity \
OPENSHELL_CONFORMANCE_BIN="$PWD/target/debug/openshell-conformance" \
mise exec -- bash plans/3835-with-docker-gateway-macos.sh \
  bash e2e/rust/e2e-docker.sh __openshell_run_docker_e2e

As in the earlier evidence, the ignored launcher copy uses host.docker.internal for the supervisor endpoint and binds the temporary mTLS gateway on all interfaces so Docker Desktop containers can reach the macOS host. This launcher adjustment is excluded from the PR. The launcher's optional static-linkage check was skipped because readelf is unavailable on this Mac. Kubernetes rescheduling and other driver E2E lanes were not run locally.

[github-actions]

Label test:e2e applied, but pull-request/4094 does not exist yet. A maintainer needs to comment /ok to test a645d9fd5d61993ed8c8f3d848955e38f0e81be5 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[drew]

/ok to test a645d9f

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997. I checked your cleanup update against the prior failed-create finding: post-commit identity failures and driver-create rejections now share compensation that removes the sandbox and key when possible, retains a durable Deleting record when cleanup must retry, and continues cleanup after caller cancellation. The added regressions cover the requested credential-store and key-deletion failures. No blocking code-review findings remain.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-e5de0a89-01: resolved by this head; the Gator-owned thread has been closed

Required test dispatch is not complete yet. Gator applied test:e2e and posted /ok to test for the current head; the contributor mirror must be created before the E2E label can be re-applied and the required workflows confirmed queued.

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: sandbox identity behavior is documented under docs/; no additional docs change is required for this cleanup-only delta
• Checks: DCO is green; required current-head branch workflows are awaiting contributor-mirror dispatch
• E2E: test:e2e is required; mirror creation was requested with /ok to test a645d9fd5d61993ed8c8f3d848955e38f0e81be5 and workflow queue confirmation remains pending
• Head SHA: a645d9fd5d61993ed8c8f3d848955e38f0e81be5
• Base SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2
• Merge base SHA: 021400be8af471f8669369e679de3e18cf0bd672
• Patch ID: 3b6a5b4b21488f22e5a5a68154f7db415e0e635c
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: e5de0a892813408bf08a36d93ab33b66a9929d18
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

/ok to test a645d9f