Skip to content

fix(supervisor): give each boundary outage its own wait recovery window - #5526

Merged
TaylorMutch merged 2 commits into
ai-openshell:mainfrom
mr-miles:fix/wait-recovery-window-reset
Nov 16, 2026
Merged

TaylorMutch merged 2 commits into
ai-openshell:mainfrom
mr-miles:fix/wait-recovery-window-reset

Conversation

@mr-miles

Copy link
Copy Markdown
Contributor

Summary

A supervisor wait() survived only the first boundary transport loss in a sandbox's lifetime. call_wait_stream opened its 30s recovery window at the first failure and never reset it. After one successful reattach, any later loss found the window already expired and returned without trying to reattach. The supervisor then exited with boundary lost during wait. This PR gives each outage its own window.

Related Issue

Closes #5525

Changes

  • Replace begin_recovery_window with WaitRecoveryWindow, which tracks the recovery deadline per connection generation. A failure on a newer generation (a reattach replaced the connection) starts a fresh CONNECT_RETRY_TIMEOUT window.
  • Repeated failures on the same generation still share one bounded window. This matters because recover_after_unavailable returns Ok without reconnecting when the probe finds the connection live, and resetting on every Ok would let a live connection whose streams keep failing retry forever.
  • accept_tcp uses the same call_wait_stream path, so a second transport loss no longer permanently stops the network proxy either.
  • Port the existing wait_recovery_window_begins_at_transport_failure test. Add tests for:
    • failures on one connection sharing a window that still expires
    • a failure 43 minutes after a reattach getting a fresh window

Release Notes

Long-running sandboxes no longer exit when the supervisor loses its connection to the sandbox a second time. Each connection loss now gets a full reconnect window.

Testing

  • cargo fmt --check
  • cargo test -p openshell-sandbox-backend --lib
  • cargo clippy -p openshell-sandbox-backend --all-targets
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mr-miles
mr-miles force-pushed the fix/wait-recovery-window-reset branch from ad53ac8 to d7b9635 Compare November 12, 2026 20:27
@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test d7b9635

@TaylorMutch TaylorMutch added the test:e2e Requires end-to-end coverage label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for d7b9635. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head d7b9635. The recovery change addresses the confirmed defect in #5525, which is now accepted. Keying the deadline by connection generation preserves a bounded budget for failures on one connection and grants a fresh budget after replacement. I found no blocking correctness issue in that production logic.

Required changes:

  • Fix the new test's Clippy error noted inline: use Duration::from_mins(43).
  • Amend both commits to add a Signed-off-by footer (git commit --amend --signoff) and remove the disallowed co-author attribution footers, following AGENTS.md. The current DCO status is green, but the commits themselves do not contain the required sign-offs.

Validation on macOS arm64 with Rust 1.95.0, from an isolated snapshot:

  • cargo test --offline --locked -p openshell-sandbox-backend --lib: 111 passed, including the actual two-disconnect transport regression.
  • The same transport regression fails on the preceding test-only commit with boundary lost during wait / boundary transport replaced, confirming it detects the original bug.
  • Formatting passes.
  • cargo clippy --offline --locked -p openshell-sandbox-backend --all-targets -- -D warnings: fails on the inline duration lint.

The issue-acceptance check now passes. Enabled test:e2e and authorized testing of this exact head; branch CI and sandbox E2E results remain required before merge. No new CLI, configuration, or protocol contract needs documenting for this correction.

@mr-miles
mr-miles force-pushed the fix/wait-recovery-window-reset branch from d7b9635 to f58c5eb Compare November 16, 2026 19:55
A supervisor wait recovers from one boundary transport loss, but after
that recovery a later loss on the replacement connection is reported as
"boundary lost during wait" without any reattach attempt, once the first
outage's 30s recovery window has passed.

Add a regression test that holds a wait open across two transport
aborts more than CONNECT_RETRY_TIMEOUT apart and expects the wait to
reattach both times and return the exit status. It fails on this
commit; the following commit fixes it.

Signed-off-by: mr-miles <miles.waller@gmail.com>
call_wait_stream opened its 30s recovery window at the first transport
failure and never reset it. A wait lasts as long as the workload, so
after one successful recovery any later transport loss found the window
long expired and returned Unavailable without attempting a reattach.
BoundaryProcess::wait() then reported "boundary lost during wait" and
the supervisor exited, even though the boundary was still reachable.
accept_tcp shares the same path and stopped the proxy the same way.

Key the window by connection generation. A failure on a newer
generation means a recovery replaced the connection, so it starts a
fresh window. Repeated failures on one generation still share a single
bounded window, so a live connection whose streams keep failing does
not retry forever.

This makes wait_survives_a_second_transport_loss_after_recovery pass,
and adds unit tests for the per-generation window itself.

Signed-off-by: mr-miles <miles.waller@gmail.com>
@mr-miles
mr-miles force-pushed the fix/wait-recovery-window-reset branch from f58c5eb to b93501b Compare November 16, 2026 19:55
@mr-miles

Copy link
Copy Markdown
Contributor Author

Addressed comments. Force-push changed the head so now needs someone to post "/ok to test b93501b"

@mr-miles
mr-miles requested a review from TaylorMutch November 16, 2026 19:59
@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test b93501b

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked head b93501b. Both requested changes are addressed: the test uses Duration::from_mins(43), and both commits have sign-off footers with the prohibited attribution footers removed.

Verified the complete source delta from the previously reviewed head: only the equivalent duration expression changed. Production recovery logic and the regression tests are otherwise identical.

All-target Clippy now passes with Rust 1.95.0: cargo clippy --offline --locked -p openshell-sandbox-backend --all-targets -- -D warnings. The previous review's 111 passing backend tests and failing pre-fix transport regression remain applicable; the full suite was not rerun for this equivalent duration expression.

No outstanding code-review findings. Approved; current-head branch CI and sandbox E2E must pass before merge. Authorized testing for this exact head.

@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test b93501b

@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 16, 2026
Merged via the queue into ai-openshell:main with commit d989d61 Nov 16, 2026
254 of 259 checks passed
@mr-miles
mr-miles deleted the fix/wait-recovery-window-reset branch November 18, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(supervisor): wait() survives only one boundary transport loss; the second is fatal without a reattach attempt

2 participants