Repository navigation
fix(server): keep the reconcile sweep from failing a sandbox mid-start - #5265
Conversation
The gateway's reconcile sweep applies polled snapshots without waiting for an in-progress start. During a stop/start restart, get_sandbox and list_sandboxes built the stopped workload from its list entry, so the previous run's exit was reported as ContainerExited and moved the sandbox from Starting to Error. While a start is in progress, inspect a stopped workload before reporting it. If it is running again, or still shows the exit recorded before the start, report it as ContainerStarting. A new exit, and any exit once the start has returned, is still reported. The start is tracked by a guard that clears the in-progress mark on every return path. Closes #5258 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Label |
A restart starts the workload before its exited supervisor. Watch reconciliation, start events, and polled snapshots could inspect the workload in that gap, see a running workload beside an exited supervisor, and stop it. Record the supervisor exit that the start observed. While that start is in progress, an exited supervisor with the recorded FinishedAt reports the workload as starting instead of containing it. A missing supervisor, a new supervisor exit, and any exit after the start returns are still contained. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rogress StartSandbox holds the sandbox lifecycle gate for the whole driver call, and the workload can still show its previous exit until the driver starts it. The watch path already waits on that gate for Starting sandboxes; the reconcile sweep did not, so a poll could move a restarting sandbox from Starting to Error. Skip such a sandbox and let the next tick or event report the settled state. This replaces the Podman-specific fence of polled snapshots. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Shiju <shiju@nvidia.com>
## Outcome Classic Hermes routing verification now requires a fresh host preflight before it can send a capable-provider request. Missing, inconsistent, or changed routing inputs reject verification. Ordinary gateway health remains separate from routing readiness and request results. ## Reason Provider attachment checks alone cannot establish that the installed router package, complete deployment, credentials, policy, and running gateway still match the selected routing configuration. ### Related issues Fixes #13864. Security qualification in #13865 and adaptive-flow qualification in #13866 remain separate. ## Changes - Add request-free validation through the pinned Hermes, Relay, and Switchyard contracts, including native activation cleanup. The existing judge qualifier runs this validation before its separate model request. - Gate onboarding, connect readiness, and status with fresh provider, role, policy, credential-reference, configuration, and gateway PID/start observations. Recheck after verification and reject changed evidence. Status and onboarding send at most one capable-provider request. - Publish the three protected routing files together only after validation and while the gateway is stopped. A filesystem lock excludes concurrent supervised startup. Existing partial or mixed files fail without automatic repair. - Preserve an expected configuration fingerprint in the gateway environment. It identifies startup inputs; it does not attest that the gateway activated routing. - Refresh the generated startup runtimes and use the production publisher in the existing routed-image fixture. Preserve existing E2E assertions, call counts, deadlines, and cleanup. The native helper is needed to use the installed loader and protected process state. A host parser cannot establish that contract. The publication lock covers concurrent startup, which per-file atomic writes cannot exclude. `hermes-switchyard-preflight.test.ts` covers publication, exclusion, malformed inputs, credential references, and cleanup. `onboard/switchyard/preflight.test.ts` covers request suppression, stale evidence, and consumer wiring. The shared managed-startup bundle is also copied by `agents/pi/Dockerfile:15`. Regenerating it changes a Pi image input, so `scripts/checks/pi-qualification-receipt-refresh.mts` requires both architecture receipts and their exact authority hashes to refresh. This is a validation prerequisite of the shared bundle change. The new deployment-verification branches select on `context.switchyardEntry`, including malformed requested routing that must remain gated. ## Verification - Focused CLI and Python publication tests: passing, including 167 tests in the final publication/readiness selection and 66 tests in the final consumer/bound-capture selection. - Existing inference invocation and route-health regression suites: 188 passed. - Managed image artifact contract suite: 23 passed in a credential-free local sandbox. Its initial temporary-directory denial was corrected in the sandbox configuration. - Actual pinned native libraries on macOS arm64: 11 valid and invalid deployment cases passed, with zero model requests and successful activation cleanup. This does not qualify Linux images. - CLI TypeScript compilation, Python compilation, shell syntax, generated artifacts, and installed commit hooks: passed. - Installed publication validation and all three compiler hooks passed in a credential-free sandbox. Git transport authentication remained outside that boundary. - Cycle 1 Linux image workflow 44179672919 passed Hermes qualification, all-agent activation on AMD64/ARM64, and rootless Podman activation. Both authenticated Pi receipts were refreshed from that run; strict source parity passes. [Cycle 2 managed-image qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/44190258380/attempts/2) also passed for the latest commit, including both architectures and rootless Podman. Docker is unavailable locally. - Cycle 2 focused validation:210 unique tests passed across seven files. The affected26-test publication suite passed after correcting the local sandbox's ancestor-directory traversal policy; credential and session paths remain denied. - Local macOS startup/config-integrity fixtures had 27 special-mode failures before the changed qualifier. All 25 extracted helper functions match canonical main byte-for-byte. No assertions or permissions were weakened. - The diff contains no secrets, API keys, or credentials. No inference credential was used for local validation. - [CI](https://github.com/NVIDIA/NemoClaw/actions/runs/44190258324/attempts/2), [portable qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/44190258234), and [Staging Brev Launchable](https://github.com/NVIDIA/NemoClaw/actions/runs/44376814689) passed for this commit. The [manual E2E suite](https://github.com/NVIDIA/NemoClaw/actions/runs/44382987374/attempts/2) remains failed on the OpenShell CA restart error; other selected test jobs passed. See the scoped maintainer exception below. This is not a passing full-suite result. ## Review notes Self-review covered NVIDIA/NemoClaw commit `9fd3072f0b6555a1432bcfc47c2837bfb3ad5141` against `495896a404d77fb4673fc41dbf18d5e0fec91b2a`, including the sensitive changes under `agents/hermes/`, `src/lib/onboard/`, and `tools/mcp-tool-discovery-runtime/`. Source inspection and behavioral tests covered publication races, partial state, credential handling, cleanup, and stale verification. Cycle 1 CodeRabbit review and all nine Advisor specialists completed. The security specialist reported no introduced security finding; the accepted startup-exclusion and consumer findings are repaired in this revision. CodeRabbit passed for the latest commit. All nine latest-head Advisor specialists completed; the request for automatic partial-publication recovery was rejected because it changes the accepted fail-closed contract. Independent human code review remains pending. Public docs under `docs/**` are deferred to the repository-authorized post-merge catch-up; owning runtime and E2E guidance is included here. [Maintainer qualification exception](#14560 (comment)), posted by @rsliter with repository admin permission, applies only to commit `9fd3072f0b6555a1432bcfc47c2837bfb3ad5141` and the OpenShell corporate-CA stop/start failure. The failed `onboard-repair` / `onboard-resume` runs and aggregates remain visible. OpenShell owns the lifecycle repair, tracked in [OpenShell #4932](ai-openshell/OpenShell#4932), with [draft repair #5265](ai-openshell/OpenShell#5265). The retained evidence fits that defect but does not conclusively identify the writer in these runs. No timeout workaround, error suppression, extra retry, or dependency change is included. Other validation and independent review requirements remain in effect. Cycle 2 of at most three published revision cycles. The author will collect CI and automated-review results, then pause before another revision. Suggestions unrelated to regression risk are rejected or deferred. No merge is requested in this cycle. --- Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added preflight verification for Hermes Switchyard routing during setup, launch readiness, status checks, and inference verification. * Routing changes are published together, with startup blocked if validation fails or another gateway is active. * Added documentation explaining routing validation and how preflight results affect verification. * **Bug Fixes** * Verification now rejects results when routing evidence is missing, unavailable, or changes during a check, without modifying provider settings. * Transient inference failures are identified separately from other failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
… replica The sweep skipped a Starting sandbox only when this replica's lifecycle gate was held, so a replica that did not own the start could still apply the previous run's exit. Also skip sandboxes whose driver operation is durably pending. Deletion keeps its own pending-operation handling. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test 7142af3 |
…tles Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Check for an in-progress start only after the driver poll: the resource-version fence already rejects a poll that overlaps a start beginning earlier. Limit the durable-claim check to Starting sandboxes, drop the prune-path guard (pending operations already block pruning), restore the original watch-delay test, and cover the gap where only the local gate defers the sweep. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test c32e689 |
TaylorMutch
left a comment
There was a problem hiding this comment.
Reviewed c32e689. No blocking findings.
The post-poll guard prevents the previous runtime's exit from failing an active start: the local lifecycle gate covers the owning replica, and the durable pending-operation claim covers other replicas during the driver call. The probe does not retain the gate or wait under the local mutation lock; resource-version checks and the final CAS fence intervening changes.
The regression tests cover local and separate-replica observers, the local pre-claim window, genuine exits after settlement, and deletion-watch cleanup during a blocked poll. Verified current CI results: Rust tests, lint, formatting, and Branch E2E checks pass. I did not rerun tests locally.
The documented cross-replica window between writing Starting and claiming the operation remains a follow-up for #5237; it is pre-existing and does not block this focused fix. #4983 is closed as superseded.
Summary
The gateway's reconcile sweep no longer applies a driver poll to a sandbox whose start is in progress. A poll that landed mid-start could see the previous run's exited container and move the sandbox from
StartingtoError(ContainerExited), even though the restarted sandbox came up. This affects every driver and every gateway topology, including multi-replica gateways.Related Issue
Closes #5258
Closes #4981
Closes #4932
Flakes this fixes
This race is behind a recurring E2E flake in any test that stops and restarts a sandbox. The test fails with
ContainerExited: Container exited while waiting for Ready, and a rerun on the same commit usually passes. The sweep runs every 60 seconds and a restart takes about 0.5 seconds, so roughly 1 in 100 restarts is exposed.Affected tests:
ssh::ssh_host_identity_stable_across_stop_start_and_new_on_recreatelifecycle::restart_stopped_sandboxpersistent_workspace::persistent_workspace_survives_stop_startStats
The last four CI flake censuses counted Podman
startfailures withContainerExited:mainEvery rerun of an affected job passed on the same commit, which is what marks this as a flake rather than a regression. The early censuses grouped failures by log signature. The six jobs below were each checked against the gateway log and match this race. Docker failures are not in the table, because the censuses counted them separately.
Confirmed instances
The November 17–19 census found five instances on Podman, and one more appeared on Docker. In each, the reconcile sweep (
reconcile{otel.name="reconcile.sandboxes"}) loggedStarting → Errorwhile StartSandbox was still running, and StartSandbox then returned success. None came from the watch path, and none were CA, DNS, or init failures.restart_stopped_sandboxrestart_stopped_sandboxpersistent_workspace_survives_stop_startThe same race was reported earlier outside CI: #4932 (Docker, v0.0.116) and #4981. #4932 also covers a temporarily absent container during a start; the existing pending-operation check on the prune path already defers that case, on the owning replica and on others.
Root cause
StartSandboxwritesStarting, claims a durable driver operation, and holds the sandbox's lifecycle gate for the whole driver call. Until the driver restarts the workload, a poll still reports the previous exit.The two paths that apply driver snapshots treat this differently:
apply_sandbox_updatesees aStartingsandbox, waits on the lifecycle gate, and re-reads the driver before applying anything.reconcile_snapshot_sandboxpolled the driver and applied the result under only the local mutation lock. Its resource-version and timestamp checks do not exclude a start that began before the sweep.A sweep tick inside the start window therefore applied the stale exit, and
ErrorreplacedStarting.Fix
After polling, the sweep skips a
Startingsandbox when either:try_lock_for, never waiting).A start that begins during the poll is already rejected by the existing resource-version check. The gate is probed, never held, so a Deleted watch can still dispatch driver cleanup. Once the start settles, the next tick or watch event applies the real state, so a genuine exit of the restarted workload is still recorded as
Error.The prune path is unchanged: it already skips sandboxes with a pending driver operation.
The net diff is limited to
crates/openshell-server/src/compute/mod.rsand is mostly tests. Earlier commits on this branch fenced the Podman driver instead; the final state reverts them. Squash on merge.Multi-replica gateways
Each replica runs its own reconcile sweep, and the lifecycle gate exists only in the replica that is running the start. A sweep on another replica cannot see that gate. The start's driver-operation claim is written to the shared store before the driver call, so every replica sees it. That claim is what protects a start from other replicas; the gate covers the owning replica in the moment between the
Startingwrite and the claim.Coverage:
Error.Errorwith the fix and 50/50 without it.One gap remains.
Startingand the claim are two separate writes. A sweep on another replica that runs entirely between them can still apply the previous exit. The start then aborts before calling the driver, so no healthy sandbox is left inError, but that start fails. Writing both in one update closes this; #5237 includes that change.Testing
reconcile_sweep_skips_a_sandbox_while_its_start_is_in_progresssweeps during a blocked start, from the owning replica and from a separate replica. It fails onmainfor both. After the start settles, the same sweep records the genuine exit asError.reconcile_sweep_skips_a_starting_sandbox_whose_gate_is_held_before_its_claimcovers the gate-only window.deleted_watch_during_sweep_poll_of_starting_sandbox_dispatches_cleanup: the gate probe does not suppress deletion cleanup.Errorwithout the fix and 0/50 with it, on both the owning and a separate replica. Genuine exits still reachedErrorwithin 0–56 ms.cargo test -p openshell-server --lib compute::passes;cargo fmtandcargo clippy -p openshell-server --all-targets -- -D warningsare clean.c32e68976(merged withmain) passed, including SSH host identity on Ubuntu 26.04 additional-CA Podman andpersistent_workspace_survives_stop_starton Fedora rootful RPM.Checklist