Repository navigation
fix(sandbox): bound runtime shutdown after SIGTERM - #5279
Merged
Merged
Conversation
After a termination signal the boundary kills the workload and stops serving, then drops its process runtime. The drop waited without a limit for every blocking task. The supervisor keeps an agent `Wait` exchange open for the life of the sandbox, and that exchange blocks on a condition variable that only the agent's async reaper wakes. When the runtime shutdown cancelled the reaper before it recorded the exit, the `Wait` never returned and the process stayed alive until SIGKILL, so every delete or stop took the full termination grace period. Stop the process runtime with a two-second bound on every return path from `serve`. Workload kill and reaping are unchanged. Part of #5025 Signed-off-by: divesh <dgude@nvidia.com>
github-actions
Bot
requested review from
TaylorMutch,
drew,
purp and
shiju-nv
November 4, 2026 21:47
7 of 8 tasks
Contributor
Author
|
/ok to test 0037efe |
6 tasks
shiju-nv
reviewed
Nov 5, 2026
This was referenced Nov 5, 2026
Run the child through run_isolated so a stall during setup fails the test instead of hanging it, and have the fake supervisor send SIGTERM once its exchanges are parked. Signed-off-by: divesh <dgude@nvidia.com>
Contributor
Author
|
/ok to test f339b5b |
shiju-nv
approved these changes
Nov 6, 2026
2 tasks
Contributor
|
Thanks for the fix. The follow-up for the root cause (the agent |
|
Label |
elezar
enabled auto-merge
November 6, 2026 13:25
This was referenced Nov 11, 2026
cv
added a commit
to NVIDIA/NemoClaw
that referenced
this pull request
Nov 16, 2026
Moves NemoClaw's OpenShell pins from v0.1.7 to v0.1.8: the native CLI, the gateway, supervisor and sandbox images, the gateway chart, the OpenShell crates, and the vendored SDK. Deployments built from this revision run OpenShell v0.1.8, which no longer waits out a Kubernetes sandbox pod's termination grace period on delete and renews client-credentials tokens in the CLI. ## Problem NemoClaw pins OpenShell v0.1.7 (`5d25211`). v0.1.8 (`f3d9765`, 80 commits later) is the latest release. Among its fixes: - deleting a sandbox on Kubernetes no longer waits out the pod's 30-second termination grace period (ai-openshell/OpenShell#5279), which every Kubernetes destroy and sandbox replacement paid (ai-openshell/OpenShell#5025); - the CLI renews an expired client-credentials token by repeating the grant (ai-openshell/OpenShell#4389, closing ai-openshell/OpenShell#4287), a limit our Kubernetes guide and GPU autoscaling example documented; - deleted sandboxes no longer leave memory behind on the gateway (ai-openshell/OpenShell#5428). ## Change - **Pins:** the Cargo revision of `openshell-core`, `openshell-sdk` and `openshell-policy`, and of the vendored SDK's `openshell-core` and `openshell-crypto`; in `versions.json`, the version, revision, Linux CLI archives, gateway, supervisor and sandbox images, and gateway chart; the managed-gateway reference, a contract test and the gateway contract module, which carry the gateway digest. The CLI checksums come from the release's `openshell-checksums-sha256.txt` and match the downloaded archives; the image and chart digests come from GHCR's `0.1.8` tags, and GHCR's `0.1.7` tags match the previous pins. - **Vendored SDK:** upstream changed only `src/client.rs` (a new `auth_interceptor()` accessor), copied unchanged. The SDK manifest still enables core telemetry, so the manifest patch stays; `NOTICE.md` records the refresh. `Cargo.lock` changes only the five OpenShell crates' revision. - **SDK integration limits:** the table in `internal-docs/design/architecture.md` still holds at v0.1.8 and now cites that revision. - **`websocket_credential_rewrite`:** OpenShell now rejects it on a `rest` endpoint (ai-openshell/OpenShell#5336), through the `openshell-policy` validation NemoClaw already runs on explicit policies. The field's description says it needs `protocol: websocket`; the JSON Schema, configuration reference and provider schema snapshot are regenerated. - **Live / Kind budgets:** with v0.1.8 the two kind cases take about 35-40 s less each: 98-108 s and 62-70 s on this pull request's runs, against 138-143 s and 94-100 s on recent `v1` runs. The per-test budget and the `live-kind` slow-timeout drop from 180 s to 140 s, and the step's wall budget from 300 s to 240 s. - **Limits removed:** the Kubernetes guide and the GPU autoscaling example no longer tell readers to sign in again when a client-credentials token expires; they say to keep `OPENSHELL_OIDC_CLIENT_SECRET` set. - **Release notes** list the new pin, the fixes above, and the policy changes that can stop an existing explicit policy: `websocket_credential_rewrite` on `rest`, filesystem paths through a symlink in a writable workspace (ai-openshell/OpenShell#5278), and IPv4-mapped IPv6 endpoint hosts (ai-openshell/OpenShell#5101). Resolves #13959. ## Upgrade notes checked - **Static credential placement** (ai-openshell/OpenShell#4886): every profile NemoClaw registers (inference, `api` integrations, Brave and Tavily search) already declares `auth_style` and `header_name`, including Anthropic's `x-api-key` and Brave's `X-Subscription-Token`, so their credentials keep resolving. - **OCSF attribution** (ai-openshell/OpenShell#5048): NemoClaw reads no OCSF records. - **SDK moves:** the Python and Go SDK changes don't apply; NemoClaw uses the Rust SDK. - **Gateway interceptors** (ai-openshell/OpenShell#5173): NemoClaw registers none. - **Chart:** v0.1.8 adds optional `metrics.tls`, `priorityClassName` and `topologySpreadConstraints`; NemoClaw sets none of them. ai-openshell/OpenShell#5478, which would replace the `metricsPort: 0` setting, isn't in v0.1.8, so that setting stays. - **User guide:** apart from the lines above, `docs/` still describes NemoClaw `637bdb6` and its OpenShell 0.1.7 links; moving them belongs to the next guide refresh. --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After SIGTERM,
openshell-sandboxcould stay alive until the container runtime SIGKILLed it, so every Kubernetes sandbox delete waited out the pod's 30 s termination grace period. The boundary now stops its process runtime within two seconds after it stops serving.Related Issue
Part of #5025. This PR fixes the
openshell-sandboxhalf; #5025 stays open for the supervisor pod, which can still wait out the grace period.Changes
run_boundaryserves throughserve_then_stop, which callsshutdown_timeoutwith a 2 s bound afterservereturns, on every return path. Dropping the runtime used to wait for blocking tasks with no limit.servereturns.sigterm_exits_while_the_supervisor_holds_an_exchange_open. It runs the boundary in a child process behind a fake supervisor that starts an agent and keeps aWaitand anAcceptNetworkexchange open, sends SIGTERM, and requires an exit within 10 s.Root cause: the supervisor keeps an agent
Waitexchange open for the life of the sandbox.ManagedProcess::waitblocks on a condition variable that only the agent's async reaper task wakes. When the runtime shutdown cancelled the reaper before it recorded the exit,Waitnever returned, and the main thread stayed inBlockingPool::shutdownunderdrop(Runtime)until SIGKILL. gdb on a hung run showed the only other runtime thread inserve_one→dispatch→BoundaryRuntime::wait→ManagedProcess::wait.Release Notes
Deleting a sandbox on Kubernetes no longer waits out the pod's 30-second termination grace period.
Testing
cargo clippy -p openshell-sandbox --all-targets -- -D warnings(also with--features perf-harness),mise run rust:format:check, andcargo test -p openshell-sandbox(290 tests) passManual, on k3s with the Kubernetes driver (30 s grace period, a supervisor pod per sandbox), changing only the sandbox runtime image:
openshell sandbox deleteopenshell sandbox stopStop was already fast in this setup because the driver stops the supervisor pod first.
Follow-ups:
Waituses the full 2 s bound every time. Waiting up to the existing 2 s reap timeout for the killed agent beforeservereturns, or recording the exit outside the cancellable reaper, would make the exit immediate.sandbox delete.Checklist