Skip to content

fix(supervisor): bound dedicated SIGTERM shutdown and stop recovering a lost boundary - #5342

Merged
FrostGod merged 3 commits into
mainfrom
fix/5025-supervisor-sigterm-exit/FrostGod
Nov 10, 2026
Merged

FrostGod merged 3 commits into
mainfrom
fix/5025-supervisor-sigterm-exit/FrostGod

Conversation

@FrostGod

@FrostGod FrostGod commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

After SIGTERM, a dedicated supervisor could outlive the runtime's stop timeout and be SIGKILLed. It retried a lost boundary connection for up to 30s on each shutdown call, and nothing bounded the total shutdown time. On Docker, the supervisor's 5s wait for the workload equaled Docker's 5s supervisor stop timeout, so stopping a workload that ignores TERM always ended in SIGKILL, on main as well. A dedicated supervisor's shutdown is now bounded at 8s, and Docker gives the supervisor 10s to stop.

Related Issue

Closes #5025

Changes

  • openshell-supervisor: on a dedicated supervisor's own SIGTERM, stopping the instance is bounded at 8s. That covers the workload's 5s TERM grace, TerminateBoundary, the exit report, and the trace flush. Losing the boundary while stopping, or hitting the deadline, finishes shutdown instead of failing it. The workload's exit status is reported if the boundary returned one; otherwise the exit code is 143.
  • openshell-sandbox-backend: add RunningBoundary::begin_stop(). After it is called, a lost connection is reported instead of recovered. The supervisor calls it only on a dedicated supervisor's own SIGTERM.
  • openshell-driver-docker: raise the supervisor container stop timeout from 5s to 10s, so the 8s bound finishes before Docker kills the supervisor. Kubernetes (30s grace) and Podman (45s stop timeout) already allow more than 8s.
  • Hosted instances in a shared supervisor keep their existing stop behavior. Stopping one still uses the host's 20s removal grace, recovers a dropped connection, sends TerminateBoundary, and reports a stop it could not confirm as Failed.
  • Un-ignore lifecycle::delete_is_prompt_when_workload_is_slow_to_handle_term.

Behavior change: in dedicated mode, a boundary connection that drops during the 5s TERM grace is not recovered. That includes the workload's own network traffic, for example from cleanup handlers. The supervisor exits within 8s either way.

Not covered: the OCSF exporter and OTLP tracer shut down after the 8s bound, and each can take up to 10s. With a slow or unreachable telemetry collector, the supervisor can still exceed Docker's 10s, as on main.

Testing

  • cargo test -p openshell-supervisor -p openshell-sandbox-backend -p openshell-driver-docker, clippy -D warnings, and fmt pass
  • Unit tests: a stopping client fails fast on a lost boundary; a dedicated stop keeps the workload's exit status when the boundary is lost or the deadline passes, and reports 143 when no status was received; a hosted-instance stop is not bounded by the 8s deadline and still fails on a lost boundary
  • Podman 5.7.0 on Ubuntu 26.04: the un-ignored test passed 24/24 with the fix vs 9/24 on main; full conformance suite 16/16; persistent_workspace_survives_stop_start 20/20
  • Docker 29.1.3: full conformance suite 16/16; ssh_host_identity passes, and every supervisor stop ends on its own in about 5s instead of a Docker SIGKILL (24/24 stops SIGKILLed on main)
  • Kubernetes conformance lane (CI, test:e2e)

The Podman failures of persistent_workspace_survives_stop_start on this branch are #5258 (fix in #5265).

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

After SIGTERM the supervisor signals, waits for, and terminates the
boundary. When the boundary connection drops during that sequence, each
call retried reattach for up to 30s, so the supervisor outlived the
runtime's stop timeout and was killed. Once shutdown starts, report a
lost connection instead of recovering it.

Un-ignore the conformance test that covers slow-TERM deletes.

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod FrostGod added test:e2e Requires end-to-end coverage area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. labels Nov 6, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/5342 does not exist yet. A maintainer needs to comment /ok to test 235e738a5dc725d5d421ad32e5d77657ea73df6d to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@FrostGod

FrostGod commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 235e738

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down. The Podman numbers (24/24 vs 9/24) are convincing for the case where the boundary dies alongside the supervisor. I'm not convinced this fully fixes the supervisor half of #5025 yet, though.

The Docker E2E failure looks like the same bug class. docker-e2e / E2E (rust-docker) failed in ssh_host_identity_survives_restarts_and_changes_after_recreation. Every other StopSandbox in that run took 142-155 ms, including other sandboxes on the same e2e-python image. The one in this test took 5,167 ms. That's Docker's 5s stop timeout on the supervisor container (stop_container(...t(5)) in the Docker driver). Right after, the reconciler moved the sandbox to Error / ContainerExited and sandbox start failed. On Docker the driver stops the supervisor before the workload container, so the boundary is still alive during shutdown and the new stopping path shouldn't be what's slow. In other words, a supervisor still failed to exit within the runtime's stop timeout on this branch. I don't know yet whether main does the same. Could you get a baseline of that test on main?

Overall shutdown deadline. This fixes one way shutdown can hang (retrying reattach to a dead boundary), but nothing bounds the total time. Signal, the 5s wait, terminate, and TerminateBoundary can each still take up to their own timeouts. Would it be cleaner to put one deadline on the whole SIGTERM path, shorter than the runtime grace periods (Docker 5s, Podman stop timeout, K8s 30s), and abandon whatever is left when it expires? That would cover this case, whatever happened on Docker above, and future causes. The stopping flag could stay as a fast path inside it.

Comments inline on the scope of the flag and on how the supervisor handles the resulting error.

Comment thread crates/openshell-supervisor/src/lib.rs Resolved
Stopping a dedicated sandbox now finishes within 8 seconds, including the
exit report and trace flush, while the workload keeps its 5 second TERM
grace. A boundary connection lost while stopping is reported as a
terminated workload instead of failing the shutdown.

Only a dedicated supervisor's own shutdown skips boundary recovery; a
shared supervisor removing one sandbox keeps recovering as before.

The Docker driver gave the supervisor 5 seconds, the same as its wait for
the workload, so stops ended in SIGKILL. It now allows 10 seconds.

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod

FrostGod commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Baseline on main: the test passes there too (9/9), but every supervisor stop ended in a Docker SIGKILL on both main and this branch (24/24). The default agent is an interactive bash that ignores SIGTERM. The supervisor waits 5s for it, and Docker only gave the supervisor 5s, so the CI failure was that existing race going the wrong way, not something this PR introduced.

Agreed on one deadline. The whole SIGTERM path is now capped at 8s: the agent gets its 5s TERM grace, and the exit report and OTLP flush fit in what's left. I also raised Docker's supervisor stop timeout to 10s so the cap fits under it. Kubernetes (30s) and Podman (45s) already had room. Locally that's 0/17 Docker SIGKILLs, and Docker conformance passes 16/16. The stopping flag stays as the fast path inside the deadline.

@FrostGod

FrostGod commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 91b6ccf

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is much better. The overall deadline is what I was hoping for, and the main-branch baseline clears up the Docker failure. CI is green now too.

A couple of things left inline. The shared-supervisor one is the one I'd want fixed before approving. The exit-code one is small.

One non-code item: the PR description is out of date. It doesn't mention the 8s shutdown cap or the Docker supervisor stop timeout going from 5s to 10s. It's also missing the note you mentioned in the earlier thread about the stopping flag: in dedicated mode, a connection drop during the 5s TERM grace won't reconnect, and that includes the workload's own network traffic. Could you update it so reviewers and the changelog reflect what this actually changes? The title could probably mention the shutdown bound as well.

Comment thread crates/openshell-supervisor/src/lib.rs Resolved
Comment thread crates/openshell-supervisor/src/lib.rs Resolved
Comment thread crates/openshell-supervisor/README.md Resolved
…s exit

Only a dedicated supervisor's own shutdown is bounded by the 8 second
deadline and treats a lost boundary as the end of the workload. A shared
supervisor stopping one hosted instance keeps the host's removal grace and
reports a stop it could not confirm as a failure, as before.

When the boundary reports the workload's exit status before it is lost,
that status is reported rather than the terminated fallback.

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test 6482d53

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this addresses everything from my last pass. Hosted instances now keep main's unbounded stop and report a stop they couldn't confirm as a failure. The workload's real exit status is kept when the boundary returned one. The README no longer overclaims the bound. Fine with bounding the exporter shutdowns in a follow-up.

The two Podman failures on this commit (podman-e2e / rust-podman-ci and conformance-integration / fedora-podman-rootless) are both persistent_workspace_survives_stop_start going to Error with an empty ContainerExited right after start. That's #5258. The same test failed the same way in the same fedora-podman-rootless lane on #5280, which doesn't touch stop logic, and the fix (#5265) hasn't merged yet. A rerun should clear it, so I'm not holding this on it.

One thing before merging: please update the PR description and title. They still describe only the first commit. It'd be good to cover the 8s shutdown bound for dedicated supervisors, the Docker supervisor stop timeout going from 5s to 10s, hosted instances keeping their existing stop behavior, and the note about the stopping flag: in dedicated mode, a connection drop during the 5s TERM grace isn't recovered, and that includes the workload's own network traffic.

@FrostGod FrostGod changed the title fix(supervisor): stop recovering a lost boundary after SIGTERM fix(supervisor): bound dedicated SIGTERM shutdown and stop recovering a lost boundary Nov 10, 2026
@FrostGod

Copy link
Copy Markdown
Contributor Author

updated the PR and re-ran the jobs

@FrostGod
FrostGod enabled auto-merge November 10, 2026 22:05
@FrostGod
FrostGod disabled auto-merge November 10, 2026 22:05
@FrostGod
FrostGod added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 89fcc52 Nov 10, 2026
246 of 256 checks passed
@FrostGod
FrostGod deleted the fix/5025-supervisor-sigterm-exit/FrostGod branch November 10, 2026 22:27
jgarciao pushed a commit to jgarciao/OpenShell that referenced this pull request Nov 11, 2026
… a lost boundary (ai-openshell#5342)

* fix(supervisor): stop recovering a lost boundary after SIGTERM

After SIGTERM the supervisor signals, waits for, and terminates the
boundary. When the boundary connection drops during that sequence, each
call retried reattach for up to 30s, so the supervisor outlived the
runtime's stop timeout and was killed. Once shutdown starts, report a
lost connection instead of recovering it.

Un-ignore the conformance test that covers slow-TERM deletes.

Signed-off-by: divesh <dgude@nvidia.com>

* fix(supervisor): bound SIGTERM shutdown and give Docker supervisors 10s

Stopping a dedicated sandbox now finishes within 8 seconds, including the
exit report and trace flush, while the workload keeps its 5 second TERM
grace. A boundary connection lost while stopping is reported as a
terminated workload instead of failing the shutdown.

Only a dedicated supervisor's own shutdown skips boundary recovery; a
shared supervisor removing one sandbox keeps recovering as before.

The Docker driver gave the supervisor 5 seconds, the same as its wait for
the workload, so stops ended in SIGKILL. It now allows 10 seconds.

Signed-off-by: divesh <dgude@nvidia.com>

* fix(supervisor): keep shared stops unbounded and report the workload's exit

Only a dedicated supervisor's own shutdown is bounded by the 8 second
deadline and treats a lost boundary as the end of the workload. A shared
supervisor stopping one hosted instance keeps the host's removal grace and
reports a stop it could not confirm as a failure, as before.

When the boundary reports the workload's exit status before it is lost,
that status is reported rather than the terminated fallback.

Signed-off-by: divesh <dgude@nvidia.com>

---------

Signed-off-by: divesh <dgude@nvidia.com>
cv added a commit to NVIDIA/NemoClaw that referenced this pull request Nov 18, 2026
Moves NemoClaw's OpenShell pins from v0.1.8 to v0.1.9: the native CLI,
the gateway, supervisor and sandbox images, the gateway chart, the
OpenShell crates, and the vendored SDK. A published managed Kubernetes
gateway now disables the chart's metrics with `metrics.enabled: false`,
which v0.1.9 requires, and the live middleware fixture moves to
OpenShell's HTTP hooks. Deployments built from this revision run
OpenShell v0.1.9, whose chart applies `caConfigMap`'s CA only to the
OIDC issuer.

## Problem

NemoClaw pins OpenShell v0.1.8 (`f3d9765`). v0.1.9 (`ce58cf8`, 51
commits later) is the latest release, and it changes three things
NemoClaw depends on:

- The chart refuses `service.metricsPort: 0` (NVIDIA/OpenShell#5478),
which NemoClaw sets on a gateway published through `expose.service`; the
chart's exposed contract scenario fails to render with
`service.metricsPort must be an integer between 1 and 65535; to disable
metrics, set metrics.enabled=false`.
- `server.oidc.caConfigMapName` no longer sets a process-wide
`SSL_CERT_FILE` on the gateway; the chart renders
`[openshell.gateway.oidc] ca_bundle`, so the CA applies only to OIDC
discovery and JWKS requests (NVIDIA/OpenShell#3673).
- Middleware selectors with a port fail validation, and the legacy
`EvaluateHttpRequest` and `HttpResponsePreReturn` hooks are deprecated,
to be removed in 0.2.0 (NVIDIA/OpenShell#4359).

Part of #13829: it settles question 2 of [cv's
answers](#13829 (comment)),
keeping `caConfigMap` as a typed field under
`target.kubernetes.authentication`. With v0.1.8 the chart turned that
ConfigMap into the gateway's process-wide `SSL_CERT_FILE`, so a field
named for the issuer's CA also set the trust of the gateway's other
HTTPS clients; with v0.1.9 it affects only the issuer, which is what a
typed authentication field says. The rest of #13829 (pull Secrets,
Docker and Podman ports, the `openshift` target) doesn't depend on this
upgrade, and the OpenShift UID nulls stay until NVIDIA/OpenShell#5483.

## Change

- **Pins:** the Cargo revision of `openshell-core`, `openshell-sdk` and
`openshell-policy`, and of the vendored SDK's `openshell-core` and
`openshell-crypto`; in `versions.json`, the version, revision, Linux CLI
archives, gateway, supervisor and sandbox images, and gateway chart; the
managed-gateway reference, a contract test and the gateway contract
module, which carry the gateway digest. The CLI checksums come from the
release's `openshell-checksums-sha256.txt`; the image and chart digests
come from GHCR's `0.1.9` tags, and GHCR's `0.1.8` tags match the
previous pins. `Cargo.lock` changes only the five OpenShell crates'
revision.
- **Vendored SDK:** upstream changed `src/client.rs` and `src/raw.rs` (a
raw `ServiceDiscovery` client), `src/transport.rs` and
`tests/client_mock.rs`, copied unchanged. The SDK manifest still enables
core telemetry, so the manifest patch stays; `NOTICE.md` records the
refresh. The SDK integration limits in
`internal-docs/design/architecture.md` still hold at v0.1.9 and cite
that revision.
- **Published gateway metrics:** `expose.service` sets the chart's
`metrics.enabled: false` instead of `service.metricsPort: 0`, so the
NodePort Service still publishes only the gRPC port and the gateway
serves no plaintext metrics. This deletes the `metricsPort` workaround
and its comment, which #14313 left for this upgrade. The chart
contract's exposed scenario failed with the v0.1.9 chart before the
change and passes after it.
- **Middleware:** the `docker-dummy` live case's middleware fixture
implements OpenShell's `HttpHooks` service and requires the
`openshell.supervisor-middleware.http-hooks` capability, deciding at
preflight without the body; it answers the legacy `EvaluateHttpRequest`
with `UNIMPLEMENTED`. The case's registration and assertions are
unchanged. A middleware selector with a port is a new case in the SDK's
OpenShell-validation test, which failed against v0.1.8 and passes
against v0.1.9; the selector's schema description and the sandbox
network guide say hosts take no port.
- **Docs:** OpenShell links move to the v0.1.9 revision; each path and
anchor exists there, and the three proxy line ranges in the sandbox
network guide move 71 lines down to the same text. The Kubernetes guide
says the gateway trusts `caConfigMap`'s CA only for the issuer.
- **Release notes** list the new pin and the v0.1.9 changes users see,
with two breaking changes: a middleware selector with a port fails plan,
and a DNS-glob endpoint no longer admits a private IP literal on its own
(NVIDIA/OpenShell#5522).

## Upgrade notes checked

- **Supervisor shutdown** (NVIDIA/OpenShell#5342): a dedicated
supervisor now stops within 8 s and Docker waits 10 s, so stops no
longer end in `SIGKILL`, and a connection that drops during shutdown is
no longer recovered. That can shorten the 13 s between stop and start in
#13953's failure, but the failure itself, a restarted supervisor finding
its sandbox session terminal, isn't addressed, NVIDIA/OpenShell#4910 is
still open, and #13953 didn't reproduce in 260 runs; #13953 stays open.
- **OpenShift UIDs:** NVIDIA/OpenShell#5483 is still open and not in
v0.1.9, so `openshift` keeps nulling the chart's UIDs (#13828).
- **OCSF** (NVIDIA/OpenShell#5606): NemoClaw reads no OCSF records; the
release notes tell log consumers how to select denials now.
- **Go SDK move, mutation admission limits, startup windows, service
discovery:** NemoClaw uses the Rust SDK and OpenShell's defaults;
authored gateway configuration can set the new keys.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period

2 participants