Skip to content

fix(sandbox): bound runtime shutdown after SIGTERM - #5279

Merged
elezar merged 2 commits into
mainfrom
fix/5025-sandbox-sigterm-exit/FrostGod
Nov 6, 2026
Merged

elezar merged 2 commits into
mainfrom
fix/5025-sandbox-sigterm-exit/FrostGod

Conversation

@FrostGod

@FrostGod FrostGod commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

After SIGTERM, openshell-sandbox could stay alive until the container runtime SIGKILLed it, so every Kubernetes sandbox delete waited out the pod's 30 s termination grace period. The boundary now stops its process runtime within two seconds after it stops serving.

Related Issue

Part of #5025. This PR fixes the openshell-sandbox half; #5025 stays open for the supervisor pod, which can still wait out the grace period.

Changes

  • run_boundary serves through serve_then_stop, which calls shutdown_timeout with a 2 s bound after serve returns, on every return path. Dropping the runtime used to wait for blocking tasks with no limit.
  • Workload kill and reaping are unchanged: termination still SIGKILLs the registered process groups before serve returns.
  • Added sigterm_exits_while_the_supervisor_holds_an_exchange_open. It runs the boundary in a child process behind a fake supervisor that starts an agent and keeps a Wait and an AcceptNetwork exchange open, sends SIGTERM, and requires an exit within 10 s.

Root cause: the supervisor keeps an agent Wait exchange open for the life of the sandbox. ManagedProcess::wait blocks on a condition variable that only the agent's async reaper task wakes. When the runtime shutdown cancelled the reaper before it recorded the exit, Wait never returned, and the main thread stayed in BlockingPool::shutdown under drop(Runtime) until SIGKILL. gdb on a hung run showed the only other runtime thread in serve_one → dispatch → BoundaryRuntime::wait → ManagedProcess::wait.

Release Notes

Deleting a sandbox on Kubernetes no longer waits out the pod's 30-second termination grace period.

Testing

  • On Linux (aarch64, Ubuntu 24.04): cargo clippy -p openshell-sandbox --all-targets -- -D warnings (also with --features perf-harness), mise run rust:format:check, and cargo test -p openshell-sandbox (290 tests) pass
  • Unit test added: fails 5 of 5 runs without the fix ("still running 10.0s after SIGTERM") and passes 20 of 20 with it (about 3.1 s)
  • E2E tests added/updated (not applicable)

Manual, on k3s with the Kubernetes driver (30 s grace period, a supervisor pod per sandbox), changing only the sandbox runtime image:

main this branch
openshell sandbox delete 32.4 s, 33.0 s, 32.0 s 6.3 s, 3.8 s, 4.1 s
agent container exit SIGKILL (137) at ~31 s exits on its own at ~3 s
openshell sandbox stop 3.3 s 4.6 s

Stop was already fast in this setup because the driver stops the supervisor pod first.

Follow-ups:

  • On Kubernetes the stuck Wait uses the full 2 s bound every time. Waiting up to the existing 2 s reap timeout for the killed agent before serve returns, or recording the exit outside the cancellable reaper, would make the exit immediate.
  • The supervisor pod still sometimes ignores SIGTERM and waits out the grace period (2 of 3 deletes above), the supervisor half of bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025. It no longer delays sandbox delete.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (not applicable)

After a termination signal the boundary kills the workload and stops
serving, then drops its process runtime. The drop waited without a limit
for every blocking task. The supervisor keeps an agent `Wait` exchange
open for the life of the sandbox, and that exchange blocks on a condition
variable that only the agent's async reaper wakes. When the runtime
shutdown cancelled the reaper before it recorded the exit, the `Wait`
never returned and the process stayed alive until SIGKILL, so every
delete or stop took the full termination grace period.

Stop the process runtime with a two-second bound on every return path
from `serve`. Workload kill and reaping are unchanged.

Part of #5025

Signed-off-by: divesh <dgude@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@FrostGod FrostGod added retired:area:sandbox Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. and removed retired:area:sandbox Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. labels Nov 4, 2026
@FrostGod FrostGod self-assigned this Nov 4, 2026
@FrostGod

FrostGod commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0037efe

Run the child through run_isolated so a stall during setup fails the test instead of hanging it, and have the fake supervisor send SIGTERM once its exchanges are parked.

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod

FrostGod commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f339b5b

@elezar

elezar commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Thanks for the fix. The follow-up for the root cause (the agent Wait never returning when shutdown cancels the reaper, so the boundary always uses the full 2 s bound) is now tracked in #5329. Note that #5291 already merged and expects this fix, so landing this PR restores the Kubernetes lifecycle conformance test.

@elezar elezar added the test:e2e Requires end-to-end coverage label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Label test:e2e applied for f339b5b. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@elezar elezar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The bounded runtime shutdown fixes the hang until SIGKILL, and the root-cause follow-up is tracked in #5329.

@elezar
elezar enabled auto-merge November 6, 2026 13:25
@elezar
elezar added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit e9ba8d9 Nov 6, 2026
179 of 181 checks passed
@elezar
elezar deleted the fix/5025-sandbox-sigterm-exit/FrostGod branch November 6, 2026 14:22
cv added a commit to NVIDIA/NemoClaw that referenced this pull request Nov 16, 2026
Moves NemoClaw's OpenShell pins from v0.1.7 to v0.1.8: the native CLI,
the gateway, supervisor and sandbox images, the gateway chart, the
OpenShell crates, and the vendored SDK. Deployments built from this
revision run OpenShell v0.1.8, which no longer waits out a Kubernetes
sandbox pod's termination grace period on delete and renews
client-credentials tokens in the CLI.

## Problem

NemoClaw pins OpenShell v0.1.7 (`5d25211`). v0.1.8 (`f3d9765`, 80
commits later) is the latest release. Among its fixes:

- deleting a sandbox on Kubernetes no longer waits out the pod's
30-second termination grace period (ai-openshell/OpenShell#5279), which every
Kubernetes destroy and sandbox replacement paid (ai-openshell/OpenShell#5025);
- the CLI renews an expired client-credentials token by repeating the
grant (ai-openshell/OpenShell#4389, closing ai-openshell/OpenShell#4287), a limit
our Kubernetes guide and GPU autoscaling example documented;
- deleted sandboxes no longer leave memory behind on the gateway
(ai-openshell/OpenShell#5428).

## Change

- **Pins:** the Cargo revision of `openshell-core`, `openshell-sdk` and
`openshell-policy`, and of the vendored SDK's `openshell-core` and
`openshell-crypto`; in `versions.json`, the version, revision, Linux CLI
archives, gateway, supervisor and sandbox images, and gateway chart; the
managed-gateway reference, a contract test and the gateway contract
module, which carry the gateway digest. The CLI checksums come from the
release's `openshell-checksums-sha256.txt` and match the downloaded
archives; the image and chart digests come from GHCR's `0.1.8` tags, and
GHCR's `0.1.7` tags match the previous pins.
- **Vendored SDK:** upstream changed only `src/client.rs` (a new
`auth_interceptor()` accessor), copied unchanged. The SDK manifest still
enables core telemetry, so the manifest patch stays; `NOTICE.md` records
the refresh. `Cargo.lock` changes only the five OpenShell crates'
revision.
- **SDK integration limits:** the table in
`internal-docs/design/architecture.md` still holds at v0.1.8 and now
cites that revision.
- **`websocket_credential_rewrite`:** OpenShell now rejects it on a
`rest` endpoint (ai-openshell/OpenShell#5336), through the `openshell-policy`
validation NemoClaw already runs on explicit policies. The field's
description says it needs `protocol: websocket`; the JSON Schema,
configuration reference and provider schema snapshot are regenerated.
- **Live / Kind budgets:** with v0.1.8 the two kind cases take about
35-40 s less each: 98-108 s and 62-70 s on this pull request's runs,
against 138-143 s and 94-100 s on recent `v1` runs. The per-test budget
and the `live-kind` slow-timeout drop from 180 s to 140 s, and the
step's wall budget from 300 s to 240 s.
- **Limits removed:** the Kubernetes guide and the GPU autoscaling
example no longer tell readers to sign in again when a
client-credentials token expires; they say to keep
`OPENSHELL_OIDC_CLIENT_SECRET` set.
- **Release notes** list the new pin, the fixes above, and the policy
changes that can stop an existing explicit policy:
`websocket_credential_rewrite` on `rest`, filesystem paths through a
symlink in a writable workspace (ai-openshell/OpenShell#5278), and IPv4-mapped
IPv6 endpoint hosts (ai-openshell/OpenShell#5101).

Resolves #13959.

## Upgrade notes checked

- **Static credential placement** (ai-openshell/OpenShell#4886): every profile
NemoClaw registers (inference, `api` integrations, Brave and Tavily
search) already declares `auth_style` and `header_name`, including
Anthropic's `x-api-key` and Brave's `X-Subscription-Token`, so their
credentials keep resolving.
- **OCSF attribution** (ai-openshell/OpenShell#5048): NemoClaw reads no OCSF
records.
- **SDK moves:** the Python and Go SDK changes don't apply; NemoClaw
uses the Rust SDK.
- **Gateway interceptors** (ai-openshell/OpenShell#5173): NemoClaw registers
none.
- **Chart:** v0.1.8 adds optional `metrics.tls`, `priorityClassName` and
`topologySpreadConstraints`; NemoClaw sets none of them.
ai-openshell/OpenShell#5478, which would replace the `metricsPort: 0` setting,
isn't in v0.1.8, so that setting stays.
- **User guide:** apart from the lines above, `docs/` still describes
NemoClaw `637bdb6` and its OpenShell 0.1.7 links; moving them belongs to
the next guide refresh.

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants