Skip to content

feat(supervisor): spread shared supervisors evenly across gateway replicas - #5436

Merged
FrostGod merged 6 commits into
mainfrom
feat/5434-supervisor-gateway-spread/FrostGod
Nov 17, 2026
Merged

FrostGod merged 6 commits into
mainfrom
feat/5434-supervisor-gateway-spread/FrostGod

Conversation

@FrostGod

Copy link
Copy Markdown
Contributor

Explainer for reviewers: open the one-page visual (gist source). It covers current vs. proposed behavior, the cap, the stay flag, rollouts, and the one trade-off.

Screenshot

Ring with a cap explainer

Summary

Shared supervisors now spread evenly across gateway replicas. A supervisor that comes in through the Service goes to the first gateway clockwise on the existing gateway ring that serves fewer than ⌈supervisors ÷ gateways⌉, and a stopping gateway hands its supervisors off the same way. Supervisors move only when they reconnect through the Service; there is no rebalancer.

Related Issue

Closes #5434

Changes

  • Protocol: HostGatewayMessage.redirect (HostRedirect { peer_endpoint }) and HostHello.stay.
  • Gateway:
    • On attach, a hello without stay is redirected, before any session starts, when the ring and cap place the member elsewhere.
    • On shutdown, each stream ends with a redirect computed on the ring without this replica.
    • Counts come from the pool member rows the reconciler already reads (MemberView.gateway_replica).
    • New helper: GatewayRing::first_below.
  • Supervisor:
    • Follows a redirect at once, without backoff, with stay.
    • Also sets stay on the sticky redial (token refresh, blips) and on the Service fallback after a failed redirect, so a stale ring cannot loop it.
    • A gateway always accepts stay.
  • Metrics: openshell_server_supervisor_hosts (supervisors per gateway) and openshell_server_supervisor_host_instances_max (busiest supervisor's sandboxes). The second is a max rather than a per-supervisor label because member IDs are pod names, which would make the label set grow without bound.
  • Docs: HA page, metrics catalog, and the supervisor-control README.

Known trade-off: nothing moves after placement, so two cases can stay uneven until supervisors restart:

  • a gateway added without a rollout;
  • a supervisor rolling update with surge, where a new supervisor can't know which old one leaves next.

Gateway rollouts and crashes, and supervisor restarts, end even.

Testing

  • cargo clippy -D warnings and cargo fmt --check for openshell-server, openshell-supervisor, openshell-supervisor-control, and the e2e crate
  • Unit tests: 2,646 in openshell-server, 293 in openshell-supervisor, 12 in openshell-supervisor-control, all pass. New tests:
    • first_below skips full replicas and falls back to the owner
    • arrivals under the cap fill every replica evenly
    • spread_home ignores the member's own record and leaving members
    • the supervisor follows a redirect at once with stay, then falls back to the Service with stay
  • E2E on a local k3d cluster (2 gateways, 2 supervisors, Postgres) with this branch's images:
    • new supervisors_spread_evenly_across_gateways (restart each supervisor, then roll the gateways, and assert one supervisor per gateway): passed 3 of 3 runs
    • pool_sessions_follow_their_supervisors_gateway: passed
    • losing_a_supervisor_pod_moves_its_sandboxes_without_restarting_them stops at its first host-endpoint call with policy_dns_trusted_gateway_unavailable, before any supervisor or gateway moves. The hand-built cluster lacks the host-gateway DNS setup the CI lane provides.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

…licas (closes #5434)

Signed-off-by: divesh <dgude@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test 0f34dee

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. rust-sdk — No explicit risk acceptance found

Under openshell_sdk::raw::proto::supervisor_host::v1, HostHello gains stay and host_gateway_message::Payload gains Redirect, invalidating formerly complete literals and exhaustive matches.

  • Impact: Raw SDK consumers constructing HostHello or exhaustively matching Payload fail to compile, although ordinary RPC calls and generated server-trait signatures remain unchanged.
  • Action: Preserve the baseline exported Rust shapes for v0.1.8 by isolating the extended private protocol in separate types, or defer the break to a permitted minor train with guidance for stay/default initialization and Redirect handling.
  • PR: feat(supervisor): spread shared supervisors evenly across gateway replicas #5436
Review details
  • Execution: complete
  • Candidate: PR #5436 (44164f0c877a9626ae9c59be58688ec1fdf2efb3)
  • Baseline: 4f79018659447c6589dc6a340fda44712ae8ef16 (4f79018659447c6589dc6a340fda44712ae8ef16)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@FrostGod
FrostGod requested a review from TaylorMutch November 10, 2026 06:11
@FrostGod FrostGod self-assigned this Nov 10, 2026
@FrostGod
FrostGod requested a review from pimlock November 10, 2026 16:55
Signed-off-by: divesh <dgude@nvidia.com>
@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test fb1e825

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two correctness issues in the placement and redirect handling; details are attached inline.

Validation: ran 12 existing ring tests and two isolated reproductions using the PR's placement functions. All passed, confirming that concurrent arrivals and handoffs can exceed the intended cap. I did not run the full build or Kubernetes E2E.

…s after a redirect

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test 9e93a2c

…at refuse redirects

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test fed4dae

@purp purp added area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:kubernetes Kubernetes operators, CRDs, deployment controllers, and native Kubernetes API integration. topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export. topic:performance Startup time, latency, throughput, overhead, scalability, benchmarking, and rate limiting. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. labels Nov 15, 2026
Signed-off-by: divesh <dgude@nvidia.com>
…gateway-spread/FrostGod

Signed-off-by: divesh <dgude@nvidia.com>

# Conflicts:
#	crates/openshell-server/src/lib.rs
@TaylorMutch

Copy link
Copy Markdown
Contributor

/ok to test bcc6448

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked at bcc6448. No remaining code blockers. The earlier shutdown-handoff issue is fixed: acceptance is preserved and resets the redirect-hop budget before a later handoff. Every local acceptance now counts in placement state, including hosts that refuse redirects.

I accept the documented best-effort placement tradeoff in place of the strict cross-gateway cap requested in my earlier comment. The interleaved-arrival sequence still produces 2:1:0; the new regression test explicitly records that behavior. Independent gateway views can leave persistent skew, and there is no background rebalance. The reported approximately 20% simulation result should be understood as an observation, not a guaranteed bound.

Validation: read the current diff and relevant controller/reconnect context; ran all 18 ring/spread tests successfully in an isolated Cargo harness using the unmodified current PR source and sha2 0.10.9, with only the surrounding LiveMember DTO and MEMBER_TTL dependency stubbed. I did not run the full server/supervisor suites or Kubernetes E2E. Branch Checks are still pending, so this approval is codewise.

@FrostGod
FrostGod added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 0be63c5 Nov 17, 2026
93 checks passed
@FrostGod
FrostGod deleted the feat/5434-supervisor-gateway-spread/FrostGod branch November 17, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:kubernetes Kubernetes operators, CRDs, deployment controllers, and native Kubernetes API integration. topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export. topic:performance Startup time, latency, throughput, overhead, scalability, benchmarking, and rate limiting. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: spread shared supervisors evenly across gateway replicas

3 participants