Skip to content

feat(tracing): show boundary connection time during sandbox startup - #5243

Merged
shiju-nv merged 4 commits into
ai-openshell:mainfrom
shiju-nv:feat/5160-boundary-connect-span/shiju-nv
Nov 20, 2026
Merged

shiju-nv merged 4 commits into
ai-openshell:mainfrom
shiju-nv:feat/5160-boundary-connect-span/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Startup traces currently charge backend connection and authentication waits to policy discovery. Add a supervisor.boundary.connect child span so operators can see that wait separately.

Related Issue

Closes #5160.

Changes

  • Trace the shared connection path for dial, backend-provided, runtime-initiated and routed transports.
  • Record openshell.boundary.transport, openshell.boundary.connect_attempts and completion status. Cancellation leaves status unset. Preserve the discovery parent; cached reuse creates no extra connection span.
  • Extend the two focused tests for authenticated waits, retries within one span, cached reuse, parent relationships, bounded fields, status, errors and cancellation. Use TCP nodelay helpers in the fixtures.
  • Document the span and its attributes in the tracing guide.

Release Notes

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)
  • Two exact runtime::tests::connection_tracing library tests: both tests passed on the rebased candidate using real TLS/RPC fixtures; the backend case fails twice before connecting in one span.
  • cargo clippy --locked -p openshell-sandbox-backend --all-targets -- -D warnings: passed on the final candidate.
  • cargo fmt -p openshell-sandbox-backend -- --check and git diff --check: passed.

A fresh Kubernetes OTLP startup capture was not performed locally.

Checklist

@shiju-nv shiju-nv added area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. test:e2e Requires end-to-end coverage topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export. labels Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied for b3d63c4. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@HarryMoss

HarryMoss commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Drafted by Codex; checked by @HarryMoss.

Thanks @shiju-nv. This looks like a useful way to address the startup timing confusion @rhuss described in #5160.

I checked b3d63c4 locally. Both connection-span tests passed, covering authenticated waits, parent relationships, cached reuse and cancellation.

One small integration point: main’s #5280 now disallows raw TcpStream::connect calls. The new test file adds two at lines 109 and 142. Could those use openshell_core::net::connect_tcp_nodelay_best_effort to satisfy the current Clippy rule?

I haven’t captured a fresh Kubernetes startup trace, so that E2E acceptance criterion remains unverified by this review.

@purp purp added meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. and removed meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. labels Nov 15, 2026

@rhuss rhuss left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc-review Summary

What Went Well

  • crates/openshell-sandbox-backend/src/runtime.rs BoundaryTransport::connect: one #[tracing::instrument] on the single physical-connect function covers the dial, backend, runtime-initiated and routed transports, which is what #5160 asks for. Instrumenting the whole async fn keeps the dial retry loop and the runtime-initiated and routed timeout waits inside one span.
  • crates/openshell-sandbox-backend/src/runtime.rs:159-168: skip_all keeps the SandboxRuntimeDescriptor (TLS material, endpoints) out of span fields. The transport label comes from an exhaustive match over TransportSource with no wildcard arm, so it stays one of four static strings and a new variant forces a decision at compile time.
  • build_grpc_channel in crates/openshell-sandbox-backend/src/runtime.rs: connect_with_connector awaits the connector eagerly, so the new span nests under the caller's supervisor.boundary.discover_policy span without manual context propagation.
  • crates/openshell-sandbox-backend/src/runtime/tests/connection_tracing.rs connection_span_covers_authenticated_wait_and_cached_reuse: oneshot gates hold the server-side TLS accept, so the check that the span is still open during authentication is deterministic and needs no sleeps. The test also checks that the parent is the discovery span and that a second discover_policy on the cached channel creates no extra span.
  • crates/openshell-sandbox-backend/src/runtime/tests/connection_tracing.rs connection_span_closes_on_error_and_cancellation: one poll followed by a drop checks span closure on cancellation, the error case matches the BackendError::Descriptor variant instead of message text, and exact field-set equality guards against later fields that could expose descriptor data.

Findings

Severity File Description Source
Minor crates/openshell-sandbox-backend/src/runtime.rs:159 The boundary.connect span closes the same way on success, on timeout and on a failed TLS hands... gator
Minor crates/openshell-sandbox-backend/src/runtime.rs:160 The new span is named boundary.connect with a bare transport field. docs/observability/trac... goal-alignment
Notable crates/openshell-sandbox-backend/src/runtime.rs:162 Linked issue #5160 lists the number of connect attempts as a useful span attribute "if cheap to... goal-alignment
Minor crates/openshell-sandbox-backend/src/runtime/tests/connection_tracing.rs:92 The dial and backend cases in connection_span_covers_authenticated_wait_and_cached_reuse b... test-quality

Review Details

These findings were produced by cc-review's reviewer agents and posted without a per-finding review.

  • Findings posted: 4
  • Findings reviewed and not posted: 0
  • Gate outcome: PASS
  • Participating reviewers: gator, architecture, production, goal-alignment, test-quality

@shiju-nv
shiju-nv force-pushed the feat/5160-boundary-connect-span/shiju-nv branch from b3d63c4 to b17d40f Compare November 17, 2026 12:54
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rhuss

rhuss commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Verified reviewer findings at b17d40f15bae6389f29cd063ba5849ccc63f4e74.

Please resolve the following threads:

@rhuss rhuss left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc-review found no issues at b17d40f.

Recording the attempt count on every retry appends duplicate OTLP
attributes. A connection that succeeds on the third attempt exports
four entries for the same key. A long retry loop can consume the span
attribute budget before the final count is recorded.

Track attempts for the lifetime of the connection future and record
the count once when it ends, including on cancellation. Verify the
exported span with a loopback OTLP collector, since the existing
map-based test replaces earlier values and hides duplicates.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
Policy discovery can outlast the 30-second connection retry window
while the sandbox starts. Restarting the connection span at each window
adds ERROR spans to successful startups and splits the wait operators
need to measure.

Use the configured startup budget for the initial connection and keep
the 30-second window for reattachment. Let the caller choose that
budget on every transport so sharing a routed registration between
discovery and attach does not shorten the attach window.

Emit the attempt count only when the supervisor dials or invokes a
backend connector. Inbound transports wait for an authenticated stream
and cannot count the runtime’s dials. If the startup deadline cancels
the wait, the startup span records the failure.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
@krishicks

Copy link
Copy Markdown
Contributor

/ok to test 6bb1615

@shiju-nv
shiju-nv added this pull request to the merge queue Nov 20, 2026
Merged via the queue into ai-openshell:main with commit 498a0bb Nov 20, 2026
160 checks passed
@shiju-nv
shiju-nv deleted the feat/5160-boundary-connect-span/shiju-nv branch November 20, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. test:e2e Requires end-to-end coverage topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(supervisor): split the boundary wait out of supervisor.boundary.discover_policy with a boundary.connect span

5 participants