Repository navigation
Conversation
|
Label |
|
Drafted by Codex; checked by @HarryMoss. Thanks @shiju-nv. This looks like a useful way to address the startup timing confusion @rhuss described in #5160. I checked b3d63c4 locally. Both connection-span tests passed, covering authenticated waits, parent relationships, cached reuse and cancellation. One small integration point: main’s #5280 now disallows raw TcpStream::connect calls. The new test file adds two at lines 109 and 142. Could those use openshell_core::net::connect_tcp_nodelay_best_effort to satisfy the current Clippy rule? I haven’t captured a fresh Kubernetes startup trace, so that E2E acceptance criterion remains unverified by this review. |
rhuss
left a comment
There was a problem hiding this comment.
cc-review Summary
What Went Well
crates/openshell-sandbox-backend/src/runtime.rsBoundaryTransport::connect: one#[tracing::instrument]on the single physical-connect function covers the dial, backend, runtime-initiated and routed transports, which is what #5160 asks for. Instrumenting the whole async fn keeps the dial retry loop and the runtime-initiated and routed timeout waits inside one span.crates/openshell-sandbox-backend/src/runtime.rs:159-168:skip_allkeeps theSandboxRuntimeDescriptor(TLS material, endpoints) out of span fields. Thetransportlabel comes from an exhaustive match overTransportSourcewith no wildcard arm, so it stays one of four static strings and a new variant forces a decision at compile time.build_grpc_channelincrates/openshell-sandbox-backend/src/runtime.rs:connect_with_connectorawaits the connector eagerly, so the new span nests under the caller'ssupervisor.boundary.discover_policyspan without manual context propagation.crates/openshell-sandbox-backend/src/runtime/tests/connection_tracing.rsconnection_span_covers_authenticated_wait_and_cached_reuse: oneshot gates hold the server-side TLS accept, so the check that the span is still open during authentication is deterministic and needs no sleeps. The test also checks that the parent is the discovery span and that a seconddiscover_policyon the cached channel creates no extra span.crates/openshell-sandbox-backend/src/runtime/tests/connection_tracing.rsconnection_span_closes_on_error_and_cancellation: one poll followed by a drop checks span closure on cancellation, the error case matches theBackendError::Descriptorvariant instead of message text, and exact field-set equality guards against later fields that could expose descriptor data.
Findings
| Severity | File | Description | Source |
|---|---|---|---|
| Minor | crates/openshell-sandbox-backend/src/runtime.rs:159 | The boundary.connect span closes the same way on success, on timeout and on a failed TLS hands... |
gator |
| Minor | crates/openshell-sandbox-backend/src/runtime.rs:160 | The new span is named boundary.connect with a bare transport field. docs/observability/trac... |
goal-alignment |
| Notable | crates/openshell-sandbox-backend/src/runtime.rs:162 | Linked issue #5160 lists the number of connect attempts as a useful span attribute "if cheap to... | goal-alignment |
| Minor | crates/openshell-sandbox-backend/src/runtime/tests/connection_tracing.rs:92 | The dial and backend cases in connection_span_covers_authenticated_wait_and_cached_reuse b... |
test-quality |
Review Details
These findings were produced by cc-review's reviewer agents and posted without a per-finding review.
- Findings posted: 4
- Findings reviewed and not posted: 0
- Gate outcome: PASS
- Participating reviewers: gator, architecture, production, goal-alignment, test-quality
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: Shiju <shiju@nvidia.com>
b3d63c4 to
b17d40f
Compare
|
Verified reviewer findings at Please resolve the following threads:
|
rhuss
left a comment
There was a problem hiding this comment.
cc-review found no issues at b17d40f.
Recording the attempt count on every retry appends duplicate OTLP attributes. A connection that succeeds on the third attempt exports four entries for the same key. A long retry loop can consume the span attribute budget before the final count is recorded. Track attempts for the lifetime of the connection future and record the count once when it ends, including on cancellation. Verify the exported span with a loopback OTLP collector, since the existing map-based test replaces earlier values and hides duplicates. Signed-off-by: Kris Hicks <khicks@nvidia.com>
Policy discovery can outlast the 30-second connection retry window while the sandbox starts. Restarting the connection span at each window adds ERROR spans to successful startups and splits the wait operators need to measure. Use the configured startup budget for the initial connection and keep the 30-second window for reattachment. Let the caller choose that budget on every transport so sharing a routed registration between discovery and attach does not shorten the attach window. Emit the attempt count only when the supervisor dials or invokes a backend connector. Inbound transports wait for an authenticated stream and cannot count the runtime’s dials. If the startup deadline cancels the wait, the startup span records the failure. Signed-off-by: Kris Hicks <khicks@nvidia.com>
|
/ok to test 6bb1615 |
Summary
Startup traces currently charge backend connection and authentication waits to policy discovery. Add a
supervisor.boundary.connectchild span so operators can see that wait separately.Related Issue
Closes #5160.
Changes
openshell.boundary.transport,openshell.boundary.connect_attemptsand completion status. Cancellation leaves status unset. Preserve the discovery parent; cached reuse creates no extra connection span.Release Notes
Testing
runtime::tests::connection_tracinglibrary tests: both tests passed on the rebased candidate using real TLS/RPC fixtures; the backend case fails twice before connecting in one span.cargo clippy --locked -p openshell-sandbox-backend --all-targets -- -D warnings: passed on the final candidate.cargo fmt -p openshell-sandbox-backend -- --checkandgit diff --check: passed.A fresh Kubernetes OTLP startup capture was not performed locally.
Checklist