Skip to content

fix(server): tombstone deleted sandboxes in log and event buses - #5428

Merged
TaylorMutch merged 1 commit into
ai-openshell:mainfrom
EmilienM:fix/5402-sandbox-bus-tombstones/EmilienM
Nov 10, 2026
Merged

TaylorMutch merged 1 commit into
ai-openshell:mainfrom
EmilienM:fix/5402-sandbox-bus-tombstones/EmilienM

Conversation

@EmilienM

@EmilienM EmilienM commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Deleted sandboxes used to leave their log and platform-event buffers behind. The buses create a sandbox's buffers for any id, and after delete cleanup ran once, late log pushes and post-Deleted driver events re-created them. Every create/delete cycle leaked about 1.7 MiB per sandbox until the gateway was OOM-killed.

Cleanup now closes the sandbox's cursor space, which makes deletion terminal: both buses drop later publishes and subscriptions for that id.

Related Issue

Closes #5402

Changes

  • Add a terminal close() to TracingLogBus. It frees both buses' per-sandbox entries and leaves a tombstone in the shared cursor allocator, so a late publisher drops its event before allocating a tail or a broadcast channel. remove() keeps its reset semantics for the resume-cursor tests.
  • Guard subscribe() on both buses, so a subscription racing a close gets an already-closed receiver instead of re-creating the entry.
  • handle_push_sandbox_logs checks the tombstone on every batch and re-reads the sandbox record at most once per second. A missing record ends the stream with NotFound; a transient store error keeps it open.
  • Delete cleanup calls close(), which also covers gateway tracing logs, OCSF audit events, and driver platform events for the deleted sandbox.

Notes: the tombstone lives on the replica that ran delete cleanup. A stream on another replica ends on its next batch through the existence recheck, but freeing that replica's buffers is tracked in #5401. A stream ends on its next received batch, not on a server-side timer. Sandboxes in Deleting are not cut off; the tombstone fires once the record is gone. A supervisor that outlives its sandbox now gets NotFound on its next batch and reconnects; making that terminal in the supervisor is a follow-up. This touches the same functions as #5408, so whichever lands second rebases.

Release Notes

Deleted sandboxes no longer leave memory behind on the gateway that served the delete. Repeated create/delete cycles keep gateway memory flat instead of growing until the gateway is OOM-killed.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

cargo test -p openshell-server --features test-support: all suites pass. cargo clippy -p openshell-server --all-targets -- -D warnings and cargo fmt --all -- --check are clean. The new tests cover publish/subscribe after close, active receivers, close-vs-publish races, repeated create/delete cycles, and a tonic-over-TCP push stream ending NotFound after delete. The Kubernetes RSS repro from the issue was not rerun.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (not applicable)

Late publishers re-created a deleted sandbox's log and platform-event
buffers after delete cleanup had freed them, so every create/delete
cycle leaked about 1.7 MiB per sandbox on the gateway that served the
delete. Nothing freed the re-created buffers again, and gateway RSS grew
until it was OOM-killed.

Mark the sandbox's cursor space closed in the shared allocator during
delete cleanup. Both buses drop publishes for a closed space before
allocating a tail or a broadcast channel, and subscriptions return an
already-closed receiver instead of re-creating the entry. The gateway
tracing layer and driver platform events are covered by the same gate.

An open PushSandboxLogs stream now checks that tombstone on every batch
and re-reads the sandbox record at most once per second, so a replica
that did not run the delete still notices the sandbox is gone. The
handler closes the local bus entry and ends the stream with NotFound; a
transient store failure keeps the stream open and retries on the next
cadence. remove() keeps its current reset semantics for the resume-cursor
tests.

Closes ai-openshell#5402

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@EmilienM
EmilienM marked this pull request as ready for review November 9, 2026 23:53
@TaylorMutch TaylorMutch added retired:area:gateway Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export. area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. and removed retired:area:gateway Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. labels Nov 10, 2026
@TaylorMutch

Copy link
Copy Markdown
Contributor

Validated the RSS reproduction for #5402 against this PR (e305308949560cd8fdaf8731b06bca0b97ce039c) and its parent (9e004a746). The results support the targeted buffer-recreation fix: the pre-fix gateway retained progressively more memory across create/delete cycles, while the PR substantially reduced that growth and returned near its post-warm-up RSS after additional idle time.

Both runs used the Helm development environment on local ARM64 k3d (Kubernetes v1.31.5+k3s1), with one gateway per release and matching sandbox/supervisor images from the PR revision. Each run created 20 sandboxes, executed a command in each, deleted all 20, then repeated four times with 8 new sandboxes: 52 sandboxes per run, 104 total. The workload image was nvcr.io/nvidia/base/ubuntu:24.04, with an idle sleep infinity main process. Provisioning, exec, and deletion were serialized.

RSS below comes directly from the gateway process's /proc/<pid>/status, sampled 50 seconds after deletion (MiB):

Total sandboxes deleted Parent gateway PR gateway
20 94.33 67.75
28 103.00 70.41
36 109.44 73.87
44 121.78 80.99
52 128.91 76.87

After the final cycle, the PR gateway later fell to 66.51 MiB RSS while idle, compared with 67.75 MiB after the warm-up cycle. That idle decrease was largely in file-backed RSS. At the matched final 50-second samples, anonymous RSS was 104.53 MiB for the parent versus 59.40 MiB for the PR.

Both valid runs completed with zero gateway restarts, constant gateway container IDs throughout each run, and no sandbox pods remaining. The isolated cluster is now stopped.

Methodology limits:

  • An earlier concurrent PR attempt hit local load-related liveness-probe timeouts and was excluded. Both valid runs used serial provisioning and the same probe overrides (timeoutSeconds=30, failureThreshold=6 for liveness/readiness).
  • The control release had fresh storage. The PR reused its fixture storage after cleaning up the interrupted attempt, then restarted the gateway before the valid run. Both valid runs started with empty sandbox lists and fresh process-local buses.
  • The 50-second PR readings are not literally flat. This short, single-replica comparison supports the specific fix, but does not establish zero long-term growth, bounded tombstone accumulation, or HA cleanup across replicas (bug(gateway): sandbox log/event buffers are freed on only one replica #5401).

@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test e305308

@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test e305308

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit e305308. No actionable code findings.

The terminal tombstone prevents both late publications and subscriptions from recreating deleted buffers, with a consistent shared-allocator-to-bus-map lock order. Cleanup is tied to durable sandbox removal. The push-stream path preserves authorization checks before existence checks and tolerates transient store failures without treating them as deletion. Added coverage exercises closure, late publishers/subscribers, publication races, deletion integration, and stream termination.

The independent Helm/k3d RSS comparison posted above supports the targeted fix: after 52 deletions, settled RSS was 128.91 MiB for the parent versus 76.87 MiB for this PR, with the PR later returning to 66.51 MiB while idle. Cross-replica cleanup remains tracked in #5401, and the intentionally retained small tombstones are documented.

Branch Checks are still running at the time of approval. The unrelated optional Review Readiness failure is excluded from this code-review decision.

@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 10, 2026
Merged via the queue into ai-openshell:main with commit ed2d522 Nov 10, 2026
93 of 104 checks passed
cv added a commit to NVIDIA/NemoClaw that referenced this pull request Nov 16, 2026
Moves NemoClaw's OpenShell pins from v0.1.7 to v0.1.8: the native CLI,
the gateway, supervisor and sandbox images, the gateway chart, the
OpenShell crates, and the vendored SDK. Deployments built from this
revision run OpenShell v0.1.8, which no longer waits out a Kubernetes
sandbox pod's termination grace period on delete and renews
client-credentials tokens in the CLI.

## Problem

NemoClaw pins OpenShell v0.1.7 (`5d25211`). v0.1.8 (`f3d9765`, 80
commits later) is the latest release. Among its fixes:

- deleting a sandbox on Kubernetes no longer waits out the pod's
30-second termination grace period (ai-openshell/OpenShell#5279), which every
Kubernetes destroy and sandbox replacement paid (ai-openshell/OpenShell#5025);
- the CLI renews an expired client-credentials token by repeating the
grant (ai-openshell/OpenShell#4389, closing ai-openshell/OpenShell#4287), a limit
our Kubernetes guide and GPU autoscaling example documented;
- deleted sandboxes no longer leave memory behind on the gateway
(ai-openshell/OpenShell#5428).

## Change

- **Pins:** the Cargo revision of `openshell-core`, `openshell-sdk` and
`openshell-policy`, and of the vendored SDK's `openshell-core` and
`openshell-crypto`; in `versions.json`, the version, revision, Linux CLI
archives, gateway, supervisor and sandbox images, and gateway chart; the
managed-gateway reference, a contract test and the gateway contract
module, which carry the gateway digest. The CLI checksums come from the
release's `openshell-checksums-sha256.txt` and match the downloaded
archives; the image and chart digests come from GHCR's `0.1.8` tags, and
GHCR's `0.1.7` tags match the previous pins.
- **Vendored SDK:** upstream changed only `src/client.rs` (a new
`auth_interceptor()` accessor), copied unchanged. The SDK manifest still
enables core telemetry, so the manifest patch stays; `NOTICE.md` records
the refresh. `Cargo.lock` changes only the five OpenShell crates'
revision.
- **SDK integration limits:** the table in
`internal-docs/design/architecture.md` still holds at v0.1.8 and now
cites that revision.
- **`websocket_credential_rewrite`:** OpenShell now rejects it on a
`rest` endpoint (ai-openshell/OpenShell#5336), through the `openshell-policy`
validation NemoClaw already runs on explicit policies. The field's
description says it needs `protocol: websocket`; the JSON Schema,
configuration reference and provider schema snapshot are regenerated.
- **Live / Kind budgets:** with v0.1.8 the two kind cases take about
35-40 s less each: 98-108 s and 62-70 s on this pull request's runs,
against 138-143 s and 94-100 s on recent `v1` runs. The per-test budget
and the `live-kind` slow-timeout drop from 180 s to 140 s, and the
step's wall budget from 300 s to 240 s.
- **Limits removed:** the Kubernetes guide and the GPU autoscaling
example no longer tell readers to sign in again when a
client-credentials token expires; they say to keep
`OPENSHELL_OIDC_CLIENT_SECRET` set.
- **Release notes** list the new pin, the fixes above, and the policy
changes that can stop an existing explicit policy:
`websocket_credential_rewrite` on `rest`, filesystem paths through a
symlink in a writable workspace (ai-openshell/OpenShell#5278), and IPv4-mapped
IPv6 endpoint hosts (ai-openshell/OpenShell#5101).

Resolves #13959.

## Upgrade notes checked

- **Static credential placement** (ai-openshell/OpenShell#4886): every profile
NemoClaw registers (inference, `api` integrations, Brave and Tavily
search) already declares `auth_style` and `header_name`, including
Anthropic's `x-api-key` and Brave's `X-Subscription-Token`, so their
credentials keep resolving.
- **OCSF attribution** (ai-openshell/OpenShell#5048): NemoClaw reads no OCSF
records.
- **SDK moves:** the Python and Go SDK changes don't apply; NemoClaw
uses the Rust SDK.
- **Gateway interceptors** (ai-openshell/OpenShell#5173): NemoClaw registers
none.
- **Chart:** v0.1.8 adds optional `metrics.tls`, `priorityClassName` and
`topologySpreadConstraints`; NemoClaw sets none of them.
ai-openshell/OpenShell#5478, which would replace the `metricsPort: 0` setting,
isn't in v0.1.8, so that setting stays.
- **User guide:** apart from the lines above, `docs/` still describes
NemoClaw `637bdb6` and its OpenShell 0.1.7 links; moving them belongs to
the next guide refresh.

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(gateway): sandbox log/event buffers are re-created after delete and leak

2 participants