Repository navigation
fix(server): tombstone deleted sandboxes in log and event buses - #5428
TaylorMutch merged 1 commit into
Conversation
Late publishers re-created a deleted sandbox's log and platform-event buffers after delete cleanup had freed them, so every create/delete cycle leaked about 1.7 MiB per sandbox on the gateway that served the delete. Nothing freed the re-created buffers again, and gateway RSS grew until it was OOM-killed. Mark the sandbox's cursor space closed in the shared allocator during delete cleanup. Both buses drop publishes for a closed space before allocating a tail or a broadcast channel, and subscriptions return an already-closed receiver instead of re-creating the entry. The gateway tracing layer and driver platform events are covered by the same gate. An open PushSandboxLogs stream now checks that tombstone on every batch and re-reads the sandbox record at most once per second, so a replica that did not run the delete still notices the sandbox is gone. The handler closes the local bus entry and ends the stream with NotFound; a transient store failure keeps the stream open and retries on the next cadence. remove() keeps its current reset semantics for the resume-cursor tests. Closes ai-openshell#5402 Signed-off-by: Emilien Macchi <emacchi@redhat.com>
|
Validated the RSS reproduction for #5402 against this PR ( Both runs used the Helm development environment on local ARM64 k3d (Kubernetes v1.31.5+k3s1), with one gateway per release and matching sandbox/supervisor images from the PR revision. Each run created 20 sandboxes, executed a command in each, deleted all 20, then repeated four times with 8 new sandboxes: 52 sandboxes per run, 104 total. The workload image was RSS below comes directly from the gateway process's
After the final cycle, the PR gateway later fell to 66.51 MiB RSS while idle, compared with 67.75 MiB after the warm-up cycle. That idle decrease was largely in file-backed RSS. At the matched final 50-second samples, anonymous RSS was 104.53 MiB for the parent versus 59.40 MiB for the PR. Both valid runs completed with zero gateway restarts, constant gateway container IDs throughout each run, and no sandbox pods remaining. The isolated cluster is now stopped. Methodology limits:
|
|
/ok to test e305308 |
|
/ok to test e305308 |
TaylorMutch
left a comment
There was a problem hiding this comment.
Reviewed commit e305308. No actionable code findings.
The terminal tombstone prevents both late publications and subscriptions from recreating deleted buffers, with a consistent shared-allocator-to-bus-map lock order. Cleanup is tied to durable sandbox removal. The push-stream path preserves authorization checks before existence checks and tolerates transient store failures without treating them as deletion. Added coverage exercises closure, late publishers/subscribers, publication races, deletion integration, and stream termination.
The independent Helm/k3d RSS comparison posted above supports the targeted fix: after 52 deletions, settled RSS was 128.91 MiB for the parent versus 76.87 MiB for this PR, with the PR later returning to 66.51 MiB while idle. Cross-replica cleanup remains tracked in #5401, and the intentionally retained small tombstones are documented.
Branch Checks are still running at the time of approval. The unrelated optional Review Readiness failure is excluded from this code-review decision.
ed2d522
Moves NemoClaw's OpenShell pins from v0.1.7 to v0.1.8: the native CLI, the gateway, supervisor and sandbox images, the gateway chart, the OpenShell crates, and the vendored SDK. Deployments built from this revision run OpenShell v0.1.8, which no longer waits out a Kubernetes sandbox pod's termination grace period on delete and renews client-credentials tokens in the CLI. ## Problem NemoClaw pins OpenShell v0.1.7 (`5d25211`). v0.1.8 (`f3d9765`, 80 commits later) is the latest release. Among its fixes: - deleting a sandbox on Kubernetes no longer waits out the pod's 30-second termination grace period (ai-openshell/OpenShell#5279), which every Kubernetes destroy and sandbox replacement paid (ai-openshell/OpenShell#5025); - the CLI renews an expired client-credentials token by repeating the grant (ai-openshell/OpenShell#4389, closing ai-openshell/OpenShell#4287), a limit our Kubernetes guide and GPU autoscaling example documented; - deleted sandboxes no longer leave memory behind on the gateway (ai-openshell/OpenShell#5428). ## Change - **Pins:** the Cargo revision of `openshell-core`, `openshell-sdk` and `openshell-policy`, and of the vendored SDK's `openshell-core` and `openshell-crypto`; in `versions.json`, the version, revision, Linux CLI archives, gateway, supervisor and sandbox images, and gateway chart; the managed-gateway reference, a contract test and the gateway contract module, which carry the gateway digest. The CLI checksums come from the release's `openshell-checksums-sha256.txt` and match the downloaded archives; the image and chart digests come from GHCR's `0.1.8` tags, and GHCR's `0.1.7` tags match the previous pins. - **Vendored SDK:** upstream changed only `src/client.rs` (a new `auth_interceptor()` accessor), copied unchanged. The SDK manifest still enables core telemetry, so the manifest patch stays; `NOTICE.md` records the refresh. `Cargo.lock` changes only the five OpenShell crates' revision. - **SDK integration limits:** the table in `internal-docs/design/architecture.md` still holds at v0.1.8 and now cites that revision. - **`websocket_credential_rewrite`:** OpenShell now rejects it on a `rest` endpoint (ai-openshell/OpenShell#5336), through the `openshell-policy` validation NemoClaw already runs on explicit policies. The field's description says it needs `protocol: websocket`; the JSON Schema, configuration reference and provider schema snapshot are regenerated. - **Live / Kind budgets:** with v0.1.8 the two kind cases take about 35-40 s less each: 98-108 s and 62-70 s on this pull request's runs, against 138-143 s and 94-100 s on recent `v1` runs. The per-test budget and the `live-kind` slow-timeout drop from 180 s to 140 s, and the step's wall budget from 300 s to 240 s. - **Limits removed:** the Kubernetes guide and the GPU autoscaling example no longer tell readers to sign in again when a client-credentials token expires; they say to keep `OPENSHELL_OIDC_CLIENT_SECRET` set. - **Release notes** list the new pin, the fixes above, and the policy changes that can stop an existing explicit policy: `websocket_credential_rewrite` on `rest`, filesystem paths through a symlink in a writable workspace (ai-openshell/OpenShell#5278), and IPv4-mapped IPv6 endpoint hosts (ai-openshell/OpenShell#5101). Resolves #13959. ## Upgrade notes checked - **Static credential placement** (ai-openshell/OpenShell#4886): every profile NemoClaw registers (inference, `api` integrations, Brave and Tavily search) already declares `auth_style` and `header_name`, including Anthropic's `x-api-key` and Brave's `X-Subscription-Token`, so their credentials keep resolving. - **OCSF attribution** (ai-openshell/OpenShell#5048): NemoClaw reads no OCSF records. - **SDK moves:** the Python and Go SDK changes don't apply; NemoClaw uses the Rust SDK. - **Gateway interceptors** (ai-openshell/OpenShell#5173): NemoClaw registers none. - **Chart:** v0.1.8 adds optional `metrics.tls`, `priorityClassName` and `topologySpreadConstraints`; NemoClaw sets none of them. ai-openshell/OpenShell#5478, which would replace the `metricsPort: 0` setting, isn't in v0.1.8, so that setting stays. - **User guide:** apart from the lines above, `docs/` still describes NemoClaw `637bdb6` and its OpenShell 0.1.7 links; moving them belongs to the next guide refresh. --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Summary
Deleted sandboxes used to leave their log and platform-event buffers behind. The buses create a sandbox's buffers for any id, and after delete cleanup ran once, late log pushes and post-
Deleteddriver events re-created them. Every create/delete cycle leaked about 1.7 MiB per sandbox until the gateway was OOM-killed.Cleanup now closes the sandbox's cursor space, which makes deletion terminal: both buses drop later publishes and subscriptions for that id.
Related Issue
Closes #5402
Changes
close()toTracingLogBus. It frees both buses' per-sandbox entries and leaves a tombstone in the shared cursor allocator, so a late publisher drops its event before allocating a tail or a broadcast channel.remove()keeps its reset semantics for the resume-cursor tests.subscribe()on both buses, so a subscription racing a close gets an already-closed receiver instead of re-creating the entry.handle_push_sandbox_logschecks the tombstone on every batch and re-reads the sandbox record at most once per second. A missing record ends the stream withNotFound; a transient store error keeps it open.close(), which also covers gateway tracing logs, OCSF audit events, and driver platform events for the deleted sandbox.Notes: the tombstone lives on the replica that ran delete cleanup. A stream on another replica ends on its next batch through the existence recheck, but freeing that replica's buffers is tracked in #5401. A stream ends on its next received batch, not on a server-side timer. Sandboxes in
Deletingare not cut off; the tombstone fires once the record is gone. A supervisor that outlives its sandbox now getsNotFoundon its next batch and reconnects; making that terminal in the supervisor is a follow-up. This touches the same functions as #5408, so whichever lands second rebases.Release Notes
Deleted sandboxes no longer leave memory behind on the gateway that served the delete. Repeated create/delete cycles keep gateway memory flat instead of growing until the gateway is OOM-killed.
Testing
cargo test -p openshell-server --features test-support: all suites pass.cargo clippy -p openshell-server --all-targets -- -D warningsandcargo fmt --all -- --checkare clean. The new tests cover publish/subscribe after close, active receivers, close-vs-publish races, repeated create/delete cycles, and a tonic-over-TCP push stream endingNotFoundafter delete. The Kubernetes RSS repro from the issue was not rerun.Checklist