Repository navigation
fix(supervisor): give each boundary outage its own wait recovery window - #5526
Conversation
ad53ac8 to
d7b9635
Compare
|
/ok to test d7b9635 |
|
Label |
TaylorMutch
left a comment
There was a problem hiding this comment.
Reviewed head d7b9635. The recovery change addresses the confirmed defect in #5525, which is now accepted. Keying the deadline by connection generation preserves a bounded budget for failures on one connection and grants a fresh budget after replacement. I found no blocking correctness issue in that production logic.
Required changes:
- Fix the new test's Clippy error noted inline: use
Duration::from_mins(43). - Amend both commits to add a
Signed-off-byfooter (git commit --amend --signoff) and remove the disallowed co-author attribution footers, following AGENTS.md. The current DCO status is green, but the commits themselves do not contain the required sign-offs.
Validation on macOS arm64 with Rust 1.95.0, from an isolated snapshot:
cargo test --offline --locked -p openshell-sandbox-backend --lib: 111 passed, including the actual two-disconnect transport regression.- The same transport regression fails on the preceding test-only commit with
boundary lost during wait/boundary transport replaced, confirming it detects the original bug. - Formatting passes.
cargo clippy --offline --locked -p openshell-sandbox-backend --all-targets -- -D warnings: fails on the inline duration lint.
The issue-acceptance check now passes. Enabled test:e2e and authorized testing of this exact head; branch CI and sandbox E2E results remain required before merge. No new CLI, configuration, or protocol contract needs documenting for this correction.
d7b9635 to
f58c5eb
Compare
A supervisor wait recovers from one boundary transport loss, but after that recovery a later loss on the replacement connection is reported as "boundary lost during wait" without any reattach attempt, once the first outage's 30s recovery window has passed. Add a regression test that holds a wait open across two transport aborts more than CONNECT_RETRY_TIMEOUT apart and expects the wait to reattach both times and return the exit status. It fails on this commit; the following commit fixes it. Signed-off-by: mr-miles <miles.waller@gmail.com>
call_wait_stream opened its 30s recovery window at the first transport failure and never reset it. A wait lasts as long as the workload, so after one successful recovery any later transport loss found the window long expired and returned Unavailable without attempting a reattach. BoundaryProcess::wait() then reported "boundary lost during wait" and the supervisor exited, even though the boundary was still reachable. accept_tcp shares the same path and stopped the proxy the same way. Key the window by connection generation. A failure on a newer generation means a recovery replaced the connection, so it starts a fresh window. Repeated failures on one generation still share a single bounded window, so a live connection whose streams keep failing does not retry forever. This makes wait_survives_a_second_transport_loss_after_recovery pass, and adds unit tests for the per-generation window itself. Signed-off-by: mr-miles <miles.waller@gmail.com>
f58c5eb to
b93501b
Compare
|
Addressed comments. Force-push changed the head so now needs someone to post "/ok to test b93501b" |
|
/ok to test b93501b |
TaylorMutch
left a comment
There was a problem hiding this comment.
Rechecked head b93501b. Both requested changes are addressed: the test uses Duration::from_mins(43), and both commits have sign-off footers with the prohibited attribution footers removed.
Verified the complete source delta from the previously reviewed head: only the equivalent duration expression changed. Production recovery logic and the regression tests are otherwise identical.
All-target Clippy now passes with Rust 1.95.0: cargo clippy --offline --locked -p openshell-sandbox-backend --all-targets -- -D warnings. The previous review's 111 passing backend tests and failing pre-fix transport regression remain applicable; the full suite was not rerun for this equivalent duration expression.
No outstanding code-review findings. Approved; current-head branch CI and sandbox E2E must pass before merge. Authorized testing for this exact head.
|
/ok to test b93501b |
d989d61
Summary
A supervisor
wait()survived only the first boundary transport loss in a sandbox's lifetime.call_wait_streamopened its 30s recovery window at the first failure and never reset it. After one successful reattach, any later loss found the window already expired and returned without trying to reattach. The supervisor then exited withboundary lost during wait. This PR gives each outage its own window.Related Issue
Closes #5525
Changes
begin_recovery_windowwithWaitRecoveryWindow, which tracks the recovery deadline per connection generation. A failure on a newer generation (a reattach replaced the connection) starts a freshCONNECT_RETRY_TIMEOUTwindow.recover_after_unavailablereturnsOkwithout reconnecting when the probe finds the connection live, and resetting on everyOkwould let a live connection whose streams keep failing retry forever.accept_tcpuses the samecall_wait_streampath, so a second transport loss no longer permanently stops the network proxy either.wait_recovery_window_begins_at_transport_failuretest. Add tests for:Release Notes
Long-running sandboxes no longer exit when the supervisor loses its connection to the sandbox a second time. Each connection loss now gets a full reconnect window.
Testing
cargo fmt --checkcargo test -p openshell-sandbox-backend --libcargo clippy -p openshell-sandbox-backend --all-targetsChecklist