Skip to content

feat(driver-contract): specify the driver contract documents as protobuf - #5508

Open
elezar wants to merge 1 commit into
refactor/5456-driver-contract-types/elezarfrom
refactor/5456-driver-contract-format/elezar
Open

elezar wants to merge 1 commit into
refactor/5456-driver-contract-types/elezarfrom
refactor/5456-driver-contract-format/elezar

Conversation

@elezar

@elezar elezar commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Specify the runtime descriptor and the boundary configuration as protobuf messages, written as binary protobuf, so non-Rust drivers and backends can produce and read them.

Related Issue

Part of #5456. Stacked on #5488.

Changes

  • Add proto/sandbox_runtime.proto. It does not import compute_driver.proto, so it repeats ResolvedWorkloadIdentity and DriverFenceEvidence. A test in openshell-core fails if the copies drift. Merging them is a follow-up.
  • openshell-driver-contract generates the messages with ordered maps so equal documents encode to equal bytes, exposes the descriptor set, and converts between the model types and the messages, validating on the way back. openshell-driver-sdk re-exports the messages.
  • Golden protobuf JSON examples stand for the binary documents in a reviewable form; tests build the binary from them and check both directions against the legacy JSON examples.

Release Notes

None.

Testing

  • Checks appropriate to the affected code and behavior pass
    • fmt, clippy, workspace tests, rust:deny:policy, buf lint and buf breaking pass at this tip. The openshell-driver-vm tests were not run to completion: they need mke2fs, which was unavailable locally.
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)
    • Not run on this version.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@elezar elezar added the area:drivers Compute and secret drivers, host OS/architecture support, backend isolation, and lifecycle. label Nov 12, 2026
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 12, 2026
@elezar
elezar force-pushed the refactor/5456-driver-contract-format/elezar branch from d0a42b0 to 8e5ed11 Compare November 12, 2026 12:38
@elezar
elezar force-pushed the refactor/5456-driver-contract-format/elezar branch from 8e5ed11 to 5d70d82 Compare November 12, 2026 12:50
@SDAChess

Copy link
Copy Markdown
Contributor

A few tradeoff questions before this becomes the contract other drivers build against:

  1. ProtoJSON support. ProtoJSON is in the official spec, and the Go, Java, Python, C++ and TypeScript (protobuf-es) runtimes support it. Our own language is the gap: prost has no ProtoJSON support, and this PR only uses prost-reflect as a dev-dependency. What will the supervisor and sandbox use to read it in production? Either option adds a third-party runtime dependency to a security-sensitive startup path. The spec also depends on edge cases like int64 as strings, enums by name and rejecting unknown fields. Have we checked that the golden files parse the same way in Go or Python, the likely languages for an out-of-tree driver?
  2. Why not binary protobuf on disk? These files go from one machine to another, are validated strictly, and several contain secrets. Do they need to be human-readable? Binary is the one encoding every runtime supports the same way, prost included, and buf convert or protoc --decode covers debugging. Some of these documents also travel as bytes inside gRPC messages (CreateSandbox, OpenBoundaryRequest.driver_descriptor), where JSON-encoded protobuf seems odd. If the reason is readability, could we say so explicitly?
  3. Why push-based? I assume the main reason is trust bootstrap: the boundary needs a trust anchor before it can authenticate anything it fetches, and the sandbox may have no route to the driver. But could we push only a minimal bootstrap (trust anchor, gateway identity, one-use credential, endpoint) and pull the rest as typed protobuf over the existing gateway session, the way Envoy splits bootstrap and xDS? Then the file format only has to cover one small document instead of four. Part 3 of refactor: specify the boundary and supervisor-registration contracts compute drivers use #5456 already plans registration as a gRPC stream. Was a pull or hybrid model considered?

@elezar

elezar commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

3. Why push-based? [...] could we push only a minimal bootstrap [...] and pull the rest as typed protobuf over the existing gateway session

Short version: for the driver-produced documents there is no channel to pull
from at the point they are needed. For the part of the auth bundle that the
gateway owns, a pull is a reasonable follow-up, but it isn't something this PR
changes.

Where each document is needed:

  • Supervisor auth bundle. GatewayCredentials::from_auth_bundle builds the
    supervisor's gateway client from the bundle's gateway_token. The bundle is
    the credential that opens the gateway session, so it can't arrive over that
    session. Only a minimal bootstrap has to be pushed.
  • Runtime descriptor. The supervisor reads it before it can reach the
    boundary, and then passes it as OpenBoundaryRequest.driver_descriptor.
    That RPC needs the descriptor's transport, TLS client config and fence
    evidence to be established.
  • BoundaryConfig and GatewayTrust. The boundary is the listener and
    needs the gateway identity and verification keys before it can authenticate
    anything. There is no outbound channel to pull over, and the boundary may be
    unable to dial out (vsock, a VM, the Windows MXC backend), which is why
    runtime-initiated links exist.

The descriptor and boundary config also describe facts that belong to the
driver: socket and TLS file paths in the boundary's namespace, fence evidence
and claim files. The gateway can't produce them, so a pull from the gateway
would make it a relay of driver state and add a gateway round trip, with its
failure modes, to every sandbox start.

Shared supervision already works the way you describe. The registration material
travels in the assignment over the openshell.supervisor.host.v1 stream, not
as files, and part 3 of #5456 covers that.

What a hybrid could do: keep a small pushed bootstrap (trust anchor, gateway
identity, endpoint and the gateway credential) and have the supervisor fetch the
gateway-owned remainder after it has a session: the OTLP headers and TLS
material and the SSH host key. Those are the secrets that sit on disk today. That
is already the case on main; this PR gives it a different encoding and doesn't
add to it. I'd rather do that as a follow-up that changes the auth bundle's
contract than fold it into this one.

@elezar

elezar commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

1. ProtoJSON support. [...] What will the supervisor and sandbox use to read it in production? [...] Have we checked that the golden files parse the same way in Go or Python?

Fair question. Today the answer is prost-reflect.

In #5509 it is an optional dependency of openshell-driver-sdk behind the
legacy-contract feature. The supervisor and the sandbox read the documents
through DynamicMessage with unknown fields denied, and the result is converted
to the legacy types with TryFrom. So it does put a third-party reflection
library on the startup path of both.

I see three ways to deal with that:

  1. Keep prost-reflect. It already works in the workspace
    (features = ["serde"]). The cost is the dependency and the runtime
    DescriptorPool in both binaries.
  2. Generate the serde impls with pbjson-build. There is no runtime
    reflection or descriptor set, and a reader becomes
    serde_json::from_slice::<pb::BoundaryConfig>. It is maintained by InfluxData
    and compatible with prost 0.14, but it is still a third-party crate and I
    haven't run it against the golden files.
  3. Use binary protobuf. prost supports it natively, so the readers need no
    new dependency, and the cross-language question reduces to the wire format.

I would choose the third: write binary protobuf on disk. That removes the
ProtoJSON reader from the startup path. The trade-off is unknown-field handling
(your question 2): prost ignores unknown fields, so strictness would come from
the exact contract_version check and not from the decoder. That only works if
every change to the schema that alters what a reader must enforce bumps the
version, so I would add tooling that checks the version is bumped when the
schema changes in that way. I haven't designed that check yet.

On the golden files: the cross-language check has not been done yet. They are
checked against the Rust conversion and against prost-reflect, in both
directions, so they show that the legacy documents and the protobuf JSON agree.
They have not been read or written by a Go or Python runtime.

@elezar
elezar force-pushed the refactor/5456-driver-contract-format/elezar branch from 5d70d82 to 07c1910 Compare November 12, 2026 15:24
@elezar
elezar added this pull request to stack #5532 November 13, 2026 12:35
@elezar
elezar force-pushed the refactor/5456-driver-contract-format/elezar branch 2 times, most recently from 3453142 to e1708b9 Compare November 13, 2026 13:07
@purp purp added topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:extensions Extension APIs and integration points, consistent design, and external use cases. labels Nov 15, 2026
@elezar
elezar force-pushed the refactor/5456-driver-contract-format/elezar branch from e1708b9 to f1ea65f Compare November 16, 2026 12:40
@elezar elezar changed the title feat(driver-sdk): specify the driver contract documents as protobuf JSON feat(driver-contract): specify the driver contract documents as protobuf Nov 16, 2026
Add proto/sandbox_runtime.proto as the language-neutral specification of the
runtime descriptor and the boundary configuration, written as binary
protobuf. The file does not import compute_driver.proto, so it repeats
ResolvedWorkloadIdentity and DriverFenceEvidence; a test in openshell-core
fails if the copies drift.

openshell-driver-contract generates the messages with ordered maps so that
equal documents encode to equal bytes, exposes the descriptor set, and
converts between the model types and the messages, validating on the way
back. Golden protobuf JSON examples stand for the binary documents in a form
that can be reviewed; the tests build the binary from them with a dev-only
reflection helper and check both directions against the legacy examples.
openshell-driver-sdk re-exports the generated messages.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the refactor/5456-driver-contract-format/elezar branch from f1ea65f to 9f629e4 Compare November 16, 2026 15:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:drivers Compute and secret drivers, host OS/architecture support, backend isolation, and lifecycle. state:accepted A maintainer decided OpenShell should pursue this issue topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:extensions Extension APIs and integration points, consistent design, and external use cases.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants