Skip to content

fix(server): keep the reconcile sweep from failing a sandbox mid-start - #5265

Merged
matthewgrossman merged 9 commits into
mainfrom
fix/5258-podman-poll-fence/matthewgrossman
Nov 19, 2026
Merged

matthewgrossman merged 9 commits into
mainfrom
fix/5258-podman-poll-fence/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

The gateway's reconcile sweep no longer applies a driver poll to a sandbox whose start is in progress. A poll that landed mid-start could see the previous run's exited container and move the sandbox from Starting to Error (ContainerExited), even though the restarted sandbox came up. This affects every driver and every gateway topology, including multi-replica gateways.

Related Issue

Closes #5258
Closes #4981
Closes #4932

Flakes this fixes

This race is behind a recurring E2E flake in any test that stops and restarts a sandbox. The test fails with ContainerExited: Container exited while waiting for Ready, and a rerun on the same commit usually passes. The sweep runs every 60 seconds and a restart takes about 0.5 seconds, so roughly 1 in 100 restarts is exposed.

Affected tests:

  • ssh::ssh_host_identity_stable_across_stop_start_and_new_on_recreate
  • lifecycle::restart_stopped_sandbox
  • persistent_workspace::persistent_workspace_survives_stop_start

Stats

The last four CI flake censuses counted Podman start failures with ContainerExited:

Census window Red jobs / run attempts Rank among flakes Reruns that passed Podman lane failure rate On main
Oct 21 – Nov 4 20 / 20 4 10 of 10 20 of 1,312 runs (1.5%), 10 lanes 7
Nov 4 – Nov 12 17 / 16 2 10 of 10 17 of 802 runs (2.1%), 6 lanes 2
Nov 13 – Nov 17 1 / 1 10 – – 0
Nov 17 – Nov 19 5 / 5 2 2 of 2 – 0
Total 43 22 of 22 9

Every rerun of an affected job passed on the same commit, which is what marks this as a flake rather than a regression. The early censuses grouped failures by log signature. The six jobs below were each checked against the gateway log and match this race. Docker failures are not in the table, because the censuses counted them separately.

Confirmed instances

The November 17–19 census found five instances on Podman, and one more appeared on Docker. In each, the reconcile sweep (reconcile{otel.name="reconcile.sandboxes"}) logged Starting → Error while StartSandbox was still running, and StartSandbox then returned success. None came from the watch path, and none were CA, DNS, or init failures.

Job Test Error before the StartSandbox response
Ubuntu 26.04 additional-CA Podman SSH host identity 228 ms (green on same-SHA rerun)
Ubuntu 26.04 additional-CA Podman SSH host identity 166 ms (green on same-SHA rerun)
Standard Podman restart_stopped_sandbox 353 ms
External Podman restart_stopped_sandbox 183 ms
Fedora rootful RPM (artifact) persistent_workspace_survives_stop_start 260 ms
Docker external driver SSH host identity 1065 ms

The same race was reported earlier outside CI: #4932 (Docker, v0.0.116) and #4981. #4932 also covers a temporarily absent container during a start; the existing pending-operation check on the prune path already defers that case, on the owning replica and on others.

Root cause

StartSandbox writes Starting, claims a durable driver operation, and holds the sandbox's lifecycle gate for the whole driver call. Until the driver restarts the workload, a poll still reports the previous exit.

The two paths that apply driver snapshots treat this differently:

  • Watch events: apply_sandbox_update sees a Starting sandbox, waits on the lifecycle gate, and re-reads the driver before applying anything.
  • Reconcile sweep: reconcile_snapshot_sandbox polled the driver and applied the result under only the local mutation lock. Its resource-version and timestamp checks do not exclude a start that began before the sweep.

A sweep tick inside the start window therefore applied the stale exit, and Error replaced Starting.

Fix

After polling, the sweep skips a Starting sandbox when either:

  • its record has a pending driver-operation claim, or
  • this replica's lifecycle gate is held (try_lock_for, never waiting).

A start that begins during the poll is already rejected by the existing resource-version check. The gate is probed, never held, so a Deleted watch can still dispatch driver cleanup. Once the start settles, the next tick or watch event applies the real state, so a genuine exit of the restarted workload is still recorded as Error.

The prune path is unchanged: it already skips sandboxes with a pending driver operation.

The net diff is limited to crates/openshell-server/src/compute/mod.rs and is mostly tests. Earlier commits on this branch fenced the Podman driver instead; the final state reverts them. Squash on merge.

Multi-replica gateways

Each replica runs its own reconcile sweep, and the lifecycle gate exists only in the replica that is running the start. A sweep on another replica cannot see that gate. The start's driver-operation claim is written to the shared store before the driver call, so every replica sees it. That claim is what protects a start from other replicas; the gate covers the owning replica in the moment between the Starting write and the claim.

Coverage:

  • The regression test runs the sweep from the owning replica and from a second replica that shares the store but not the gate. Without the claim check, the second-replica case fails with Error.
  • In the timing model below, the second-replica variant had 0/50 spurious Error with the fix and 50/50 without it.

One gap remains. Starting and the claim are two separate writes. A sweep on another replica that runs entirely between them can still apply the previous exit. The start then aborts before calling the driver, so no healthy sandbox is left in Error, but that start fails. Writing both in one update closes this; #5237 includes that change.

Testing

  • reconcile_sweep_skips_a_sandbox_while_its_start_is_in_progress sweeps during a blocked start, from the owning replica and from a separate replica. It fails on main for both. After the start settles, the same sweep records the genuine exit as Error.
  • reconcile_sweep_skips_a_starting_sandbox_whose_gate_is_held_before_its_claim covers the gate-only window.
  • deleted_watch_during_sweep_poll_of_starting_sandbox_dispatches_cleanup: the gate probe does not suppress deletion cleanup.
  • Mutations: removing the claim check, the gate check, or both each fails a test; always skipping fails the genuine-exit assertion.
  • Timing model (300–500 ms starts, 10–50 ms sweeps, 50 trials per case): 50/50 spurious Error without the fix and 0/50 with it, on both the owning and a separate replica. Genuine exits still reached Error within 0–56 ms.
  • cargo test -p openshell-server --lib compute:: passes; cargo fmt and cargo clippy -p openshell-server --all-targets -- -D warnings are clean.
  • Branch E2E on c32e68976 (merged with main) passed, including SSH host identity on Ubuntu 26.04 additional-CA Podman and persistent_workspace_survives_stop_start on Fedora rootful RPM.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

The gateway's reconcile sweep applies polled snapshots without waiting
for an in-progress start. During a stop/start restart, get_sandbox and
list_sandboxes built the stopped workload from its list entry, so the
previous run's exit was reported as ContainerExited and moved the
sandbox from Starting to Error.

While a start is in progress, inspect a stopped workload before
reporting it. If it is running again, or still shows the exit recorded
before the start, report it as ContainerStarting. A new exit, and any
exit once the start has returned, is still reported. The start is
tracked by a guard that clears the in-progress mark on every return
path.

Closes #5258

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied for c15cece. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

A restart starts the workload before its exited supervisor. Watch
reconciliation, start events, and polled snapshots could inspect the
workload in that gap, see a running workload beside an exited
supervisor, and stop it.

Record the supervisor exit that the start observed. While that start is
in progress, an exited supervisor with the recorded FinishedAt reports
the workload as starting instead of containing it. A missing
supervisor, a new supervisor exit, and any exit after the start returns
are still contained.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rogress

StartSandbox holds the sandbox lifecycle gate for the whole driver call,
and the workload can still show its previous exit until the driver starts
it. The watch path already waits on that gate for Starting sandboxes; the
reconcile sweep did not, so a poll could move a restarting sandbox from
Starting to Error. Skip such a sandbox and let the next tick or event
report the settled state. This replaces the Podman-specific fence of
polled snapshots.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman changed the title fix(podman): keep a polled stale exit from failing an in-progress start fix(server): skip the reconcile poll of a sandbox whose start is in progress Nov 5, 2026
@matthewgrossman
matthewgrossman marked this pull request as draft November 10, 2026 05:37
@copy-pr-bot

copy-pr-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Shiju <shiju@nvidia.com>
@purp purp added area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. labels Nov 15, 2026
cjagwani pushed a commit to NVIDIA/NemoClaw that referenced this pull request Nov 18, 2026
## Outcome

Classic Hermes routing verification now requires a fresh host preflight
before it can send a capable-provider request. Missing, inconsistent, or
changed routing inputs reject verification. Ordinary gateway health
remains separate from routing readiness and request results.

## Reason

Provider attachment checks alone cannot establish that the installed
router package, complete deployment, credentials, policy, and running
gateway still match the selected routing configuration.

### Related issues

Fixes #13864.

Security qualification in #13865 and adaptive-flow qualification in
#13866 remain separate.

## Changes

- Add request-free validation through the pinned Hermes, Relay, and
Switchyard contracts, including native activation cleanup. The existing
judge qualifier runs this validation before its separate model request.
- Gate onboarding, connect readiness, and status with fresh provider,
role, policy, credential-reference, configuration, and gateway PID/start
observations. Recheck after verification and reject changed evidence.
Status and onboarding send at most one capable-provider request.
- Publish the three protected routing files together only after
validation and while the gateway is stopped. A filesystem lock excludes
concurrent supervised startup. Existing partial or mixed files fail
without automatic repair.
- Preserve an expected configuration fingerprint in the gateway
environment. It identifies startup inputs; it does not attest that the
gateway activated routing.
- Refresh the generated startup runtimes and use the production
publisher in the existing routed-image fixture. Preserve existing E2E
assertions, call counts, deadlines, and cleanup.

The native helper is needed to use the installed loader and protected
process state. A host parser cannot establish that contract. The
publication lock covers concurrent startup, which per-file atomic writes
cannot exclude. `hermes-switchyard-preflight.test.ts` covers
publication, exclusion, malformed inputs, credential references, and
cleanup. `onboard/switchyard/preflight.test.ts` covers request
suppression, stale evidence, and consumer wiring.

The shared managed-startup bundle is also copied by
`agents/pi/Dockerfile:15`. Regenerating it changes a Pi image input, so
`scripts/checks/pi-qualification-receipt-refresh.mts` requires both
architecture receipts and their exact authority hashes to refresh. This
is a validation prerequisite of the shared bundle change. The new
deployment-verification branches select on `context.switchyardEntry`,
including malformed requested routing that must remain gated.

## Verification

- Focused CLI and Python publication tests: passing, including 167 tests
in the final publication/readiness selection and 66 tests in the final
consumer/bound-capture selection.
- Existing inference invocation and route-health regression suites: 188
passed.
- Managed image artifact contract suite: 23 passed in a credential-free
local sandbox. Its initial temporary-directory denial was corrected in
the sandbox configuration.
- Actual pinned native libraries on macOS arm64: 11 valid and invalid
deployment cases passed, with zero model requests and successful
activation cleanup. This does not qualify Linux images.
- CLI TypeScript compilation, Python compilation, shell syntax,
generated artifacts, and installed commit hooks: passed.
- Installed publication validation and all three compiler hooks passed
in a credential-free sandbox. Git transport authentication remained
outside that boundary.
- Cycle 1 Linux image workflow 44179672919 passed Hermes qualification,
all-agent activation on AMD64/ARM64, and rootless Podman activation.
Both authenticated Pi receipts were refreshed from that run; strict
source parity passes. [Cycle 2 managed-image
qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/44190258380/attempts/2)
also passed for the latest commit, including both architectures and
rootless Podman. Docker is unavailable locally.
- Cycle 2 focused validation:210 unique tests passed across seven files.
The affected26-test publication suite passed after correcting the local
sandbox's ancestor-directory traversal policy; credential and session
paths remain denied.
- Local macOS startup/config-integrity fixtures had 27 special-mode
failures before the changed qualifier. All 25 extracted helper functions
match canonical main byte-for-byte. No assertions or permissions were
weakened.
- The diff contains no secrets, API keys, or credentials. No inference
credential was used for local validation.
-
[CI](https://github.com/NVIDIA/NemoClaw/actions/runs/44190258324/attempts/2),
[portable
qualification](https://github.com/NVIDIA/NemoClaw/actions/runs/44190258234),
and [Staging Brev
Launchable](https://github.com/NVIDIA/NemoClaw/actions/runs/44376814689)
passed for this commit. The [manual E2E
suite](https://github.com/NVIDIA/NemoClaw/actions/runs/44382987374/attempts/2)
remains failed on the OpenShell CA restart error; other selected test
jobs passed. See the scoped maintainer exception below. This is not a
passing full-suite result.

## Review notes

Self-review covered NVIDIA/NemoClaw commit
`9fd3072f0b6555a1432bcfc47c2837bfb3ad5141` against
`495896a404d77fb4673fc41dbf18d5e0fec91b2a`, including the sensitive
changes under `agents/hermes/`, `src/lib/onboard/`, and
`tools/mcp-tool-discovery-runtime/`. Source inspection and behavioral
tests covered publication races, partial state, credential handling,
cleanup, and stale verification. Cycle 1 CodeRabbit review and all nine
Advisor specialists completed. The security specialist reported no
introduced security finding; the accepted startup-exclusion and consumer
findings are repaired in this revision. CodeRabbit passed for the latest
commit. All nine latest-head Advisor specialists completed; the request
for automatic partial-publication recovery was rejected because it
changes the accepted fail-closed contract. Independent human code review
remains pending. Public docs under `docs/**` are deferred to the
repository-authorized post-merge catch-up; owning runtime and E2E
guidance is included here.

[Maintainer qualification
exception](#14560 (comment)),
posted by @rsliter with repository admin permission, applies only to
commit `9fd3072f0b6555a1432bcfc47c2837bfb3ad5141` and the OpenShell
corporate-CA stop/start failure. The failed `onboard-repair` /
`onboard-resume` runs and aggregates remain visible. OpenShell owns the
lifecycle repair, tracked in [OpenShell
#4932](ai-openshell/OpenShell#4932), with [draft
repair #5265](ai-openshell/OpenShell#5265). The
retained evidence fits that defect but does not conclusively identify
the writer in these runs. No timeout workaround, error suppression,
extra retry, or dependency change is included. Other validation and
independent review requirements remain in effect.

Cycle 2 of at most three published revision cycles. The author will
collect CI and automated-review results, then pause before another
revision. Suggestions unrelated to regression risk are rejected or
deferred. No merge is requested in this cycle.

---
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added preflight verification for Hermes Switchyard routing during
setup, launch readiness, status checks, and inference verification.
* Routing changes are published together, with startup blocked if
validation fails or another gateway is active.
* Added documentation explaining routing validation and how preflight
results affect verification.
* **Bug Fixes**
* Verification now rejects results when routing evidence is missing,
unavailable, or changes during a check, without modifying provider
settings.
* Transient inference failures are identified separately from other
failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
… replica

The sweep skipped a Starting sandbox only when this replica's lifecycle gate was held, so a replica that did not own the start could still apply the previous run's exit. Also skip sandboxes whose driver operation is durably pending. Deletion keeps its own pending-operation handling.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 7142af3

…tles

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Check for an in-progress start only after the driver poll: the resource-version fence already rejects a poll that overlaps a start beginning earlier. Limit the durable-claim check to Starting sandboxes, drop the prune-path guard (pending operations already block pruning), restore the original watch-delay test, and cover the gap where only the local gate defers the sweep.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test c32e689

@matthewgrossman matthewgrossman changed the title fix(server): skip the reconcile poll of a sandbox whose start is in progress fix(server): keep the reconcile sweep from failing a sandbox mid-start Nov 19, 2026
@matthewgrossman
matthewgrossman marked this pull request as ready for review November 19, 2026 19:16

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed c32e689. No blocking findings.

The post-poll guard prevents the previous runtime's exit from failing an active start: the local lifecycle gate covers the owning replica, and the durable pending-operation claim covers other replicas during the driver call. The probe does not retain the gate or wait under the local mutation lock; resource-version checks and the final CAS fence intervening changes.

The regression tests cover local and separate-replica observers, the local pre-claim window, genuine exits after settlement, and deletion-watch cleanup during a blocked poll. Verified current CI results: Rust tests, lint, formatting, and Branch E2E checks pass. I did not rerun tests locally.

The documented cross-replica window between writing Starting and claiming the operation remains a follow-up for #5237; it is pre-existing and does not block this focused fix. #4983 is closed as superseded.

@matthewgrossman
matthewgrossman added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit db7d7a6 Nov 19, 2026
168 checks passed
@matthewgrossman
matthewgrossman deleted the fix/5258-podman-poll-fence/matthewgrossman branch November 19, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. test:e2e Requires end-to-end coverage topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts.

Projects

None yet

4 participants