Repository navigation
feat(supervisor): spread shared supervisors evenly across gateway replicas - #5436
Conversation
…licas (closes #5434) Signed-off-by: divesh <dgude@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-5436.docs.buildwithfern.com/openshell |
|
/ok to test 0f34dee |
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. rust-sdk — No explicit risk acceptance foundUnder openshell_sdk::raw::proto::supervisor_host::v1, HostHello gains stay and host_gateway_message::Payload gains Redirect, invalidating formerly complete literals and exhaustive matches.
Review details
Advisory review; publication gates are unchanged. |
Signed-off-by: divesh <dgude@nvidia.com>
|
/ok to test fb1e825 |
TaylorMutch
left a comment
There was a problem hiding this comment.
I found two correctness issues in the placement and redirect handling; details are attached inline.
Validation: ran 12 existing ring tests and two isolated reproductions using the PR's placement functions. All passed, confirming that concurrent arrivals and handoffs can exceed the intended cap. I did not run the full build or Kubernetes E2E.
…s after a redirect Signed-off-by: divesh <dgude@nvidia.com>
|
/ok to test 9e93a2c |
…at refuse redirects Signed-off-by: divesh <dgude@nvidia.com>
|
/ok to test fed4dae |
Signed-off-by: divesh <dgude@nvidia.com>
…gateway-spread/FrostGod Signed-off-by: divesh <dgude@nvidia.com> # Conflicts: # crates/openshell-server/src/lib.rs
|
/ok to test bcc6448 |
TaylorMutch
left a comment
There was a problem hiding this comment.
Rechecked at bcc6448. No remaining code blockers. The earlier shutdown-handoff issue is fixed: acceptance is preserved and resets the redirect-hop budget before a later handoff. Every local acceptance now counts in placement state, including hosts that refuse redirects.
I accept the documented best-effort placement tradeoff in place of the strict cross-gateway cap requested in my earlier comment. The interleaved-arrival sequence still produces 2:1:0; the new regression test explicitly records that behavior. Independent gateway views can leave persistent skew, and there is no background rebalance. The reported approximately 20% simulation result should be understood as an observation, not a guaranteed bound.
Validation: read the current diff and relevant controller/reconnect context; ran all 18 ring/spread tests successfully in an isolated Cargo harness using the unmodified current PR source and sha2 0.10.9, with only the surrounding LiveMember DTO and MEMBER_TTL dependency stubbed. I did not run the full server/supervisor suites or Kubernetes E2E. Branch Checks are still pending, so this approval is codewise.
Summary
Shared supervisors now spread evenly across gateway replicas. A supervisor that comes in through the Service goes to the first gateway clockwise on the existing gateway ring that serves fewer than ⌈supervisors ÷ gateways⌉, and a stopping gateway hands its supervisors off the same way. Supervisors move only when they reconnect through the Service; there is no rebalancer.
Related Issue
Closes #5434
Changes
HostGatewayMessage.redirect(HostRedirect { peer_endpoint }) andHostHello.stay.stayis redirected, before any session starts, when the ring and cap place the member elsewhere.MemberView.gateway_replica).GatewayRing::first_below.stay.stayon the sticky redial (token refresh, blips) and on the Service fallback after a failed redirect, so a stale ring cannot loop it.stay.openshell_server_supervisor_hosts(supervisors per gateway) andopenshell_server_supervisor_host_instances_max(busiest supervisor's sandboxes). The second is a max rather than a per-supervisor label because member IDs are pod names, which would make the label set grow without bound.Known trade-off: nothing moves after placement, so two cases can stay uneven until supervisors restart:
Gateway rollouts and crashes, and supervisor restarts, end even.
Testing
cargo clippy -D warningsandcargo fmt --checkforopenshell-server,openshell-supervisor,openshell-supervisor-control, and the e2e crateopenshell-server, 293 inopenshell-supervisor, 12 inopenshell-supervisor-control, all pass. New tests:first_belowskips full replicas and falls back to the ownerspread_homeignores the member's own record and leaving membersstay, then falls back to the Service withstaysupervisors_spread_evenly_across_gateways(restart each supervisor, then roll the gateways, and assert one supervisor per gateway): passed 3 of 3 runspool_sessions_follow_their_supervisors_gateway: passedlosing_a_supervisor_pod_moves_its_sandboxes_without_restarting_themstops at its first host-endpoint call withpolicy_dns_trusted_gateway_unavailable, before any supervisor or gateway moves. The hand-built cluster lacks the host-gateway DNS setup the CI lane provides.Checklist