Skip to content

feat(audit): record user and operator gateway mutations as OCSF events - #5606

Merged
krishicks merged 1 commit into
mainfrom
feat/2911-record-gateway-mutations/krishicks
Nov 16, 2026
Merged

krishicks merged 1 commit into
mainfrom
feat/2911-record-gateway-mutations/krishicks

Conversation

@krishicks

@krishicks krishicks commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The gateway records authentication and authorization outcomes but not the state changes users and operators make, so nothing structured shows who created, changed, or deleted a resource, or whether the call succeeded. This PR emits one OCSF API Activity event for every call a user or mTLS operator makes to a write-scoped RPC. It is the first of a stack for #2911. Later PRs name the resources each call acted on and record no-ops.

Related Issue

Part of #2911

Changes

  • AuthGrpcRouter wraps each authenticated request in audit::with_audited_request, which emits one API Activity (6003) event when a User or Operator principal calls a cataloged method. The event carries the actor, src_endpoint, api.operation, the request ID as metadata.correlation_uid, and an outcome derived from the gRPC status:
    • OK: Success.
    • A refused request: Failure with rejected:<code> (Low).
    • A request the gateway could not complete: Failure with error:<code> (Medium).
    • A call dropped before it finished, such as when its client disconnects: Unknown with unknown:cancelled (Medium), because the change it started may still apply.
  • Exec and port forwarding stream their responses, so they are recorded when the stream ends, with the status from its trailers. A stream dropped first records unknown:cancelled.
  • An authorization denial for the call stays its only record. It now carries the method's activity_id instead of Other.
  • auth/audit/catalog.rs maps each audited method to Create, Update, Delete, or Other. A test walks the service descriptors and fails on a write-scoped method missing from the catalog, or on a stale entry.
  • Sandbox, supervisor, and gateway peer principals and read-only methods emit no record.
  • openshell-ocsf: ApiActivityBuilder accepts an activity, and ActivityId gains the API Activity values.
  • Docs: a "Gateway Mutation Records" section in docs/observability/ocsf-json-export.mdx; a note in proto/README.md on adding write-scoped RPCs to the catalog.

Before / after

Real OCSF output from the gateway auth router, called by user alice with an x-request-id:

Call → result Before (main) After
DeleteSandbox → OK no event API Activity: Delete, Success, Informational
StopSandbox → NOT_FOUND no event API Activity: Update, Failure, Low, rejected:not_found
CreateWorkspace, client disconnects first no event API Activity: Create, Unknown, Medium, unknown:cancelled
CreateProvider, missing role API Activity: Other (99) denial, missing_role API Activity: Create (1) denial, missing_role, still the only record

Each record names actor alice, src_endpoint.ip, and correlation_uid set to the request ID.

Release Notes

Gateway authorization denials of write methods now carry the method's activity (Create, Update, or Delete) instead of Other (99). Select denials by class_uid: 6003 and status_id: 2 rather than by type_uid: 600399.

Testing

  • mise run pre-commit passes
  • Unit tests added: router records for user and operator mutations, failure categories, cancellation, streams that end with a status or are dropped, handler and router denials, unaudited principals and reads, and catalog coverage of write-scoped methods
  • E2E tests added/updated (not applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

Write-RPC authorization denials change activity_id/type_uid from 99/600399 to Create 1/600301, Update 3/600303, or Delete 4/600304.

  • Impact: Existing JSONL and OTLP SIEM queries selecting type_uid=600399 silently miss affected denials, including CreateSandbox and DeleteProvider refusals.
  • Action: Preserve activity_id=99/type_uid=600399 for existing denial events in v0.1.9 and defer reclassification to a minor release with notice and SIEM-query migration guidance.
  • PR: feat(audit): record user and operator gateway mutations as OCSF events #5606
Review details
  • Execution: complete
  • Candidate: PR #5606 (008d84150a6e631a17f4b0bf5622b8bf6b6b74d7)
  • Baseline: 6070017b2b476a90d5dd64e204cb8f9ede19d985 (6070017b2b476a90d5dd64e204cb8f9ede19d985)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@krishicks

Copy link
Copy Markdown
Contributor Author

Drafted by 🤖, signed off by @krishicks

Accepting this change. The reclassification is deliberate, and the PR's release notes will give the upgrade step.

  • The documented way to select denials is unchanged: class_uid: 6003 with status_id: 2 and a reason in status_detail (missing_role, not_workspace_member, …). The docs never promised activity_id: 99 for denials.
  • 99 is OCSF's "Other", not a denial marker. type_uid=600399 also matches exec and ForwardTcp records.
  • With this PR, a denied call and an allowed call to the same method share an activity_id, so a query for "all CreateSandbox attempts" selects both. Keeping 99 for denials until a later minor release would split every per-method query in the meantime.
  • Scope: the change covers denials on the write-scoped methods in the audit catalog, for every principal type, including sandbox and gateway peer callers. Denials on other methods remain Other (99).

Upgrade step: queries that selected denials by type_uid=600399 or activity_id=99 should select class_uid=6003 AND status_id=2 and filter on status_detail instead.

@krishicks
krishicks force-pushed the feat/2911-record-gateway-mutations/krishicks branch from 35a5499 to c82b1b3 Compare November 16, 2026 17:08
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — Risk explicitly accepted

Authorization denials for cataloged Create, Update, and Delete RPCs change activity_id/type_uid from 99/600399 to 1/600301, 3/600303, or 4/600304.

  • Impact: Existing JSONL and OTLP SIEM queries filtering activity_id=99 or type_uid=600399 silently miss affected denials for every principal type, including sandbox and gateway peer callers.

  • Action: Preserve denial activity_id=99/type_uid=600399 in v0.1.9 and defer reclassification to a minor train with guidance to filter class_uid=6003, status_id=2, and authorization-specific status_detail reasons.

  • PR: feat(audit): record user and operator gateway mutations as OCSF events #5606

  • Acceptance: comment

Review details
  • Execution: complete
  • Candidate: PR #5606 (6b3934f1824fd8ef248abb8bcf45f6440849a7d4)
  • Baseline: 6070017b2b476a90d5dd64e204cb8f9ede19d985 (6070017b2b476a90d5dd64e204cb8f9ede19d985)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The centralized auditing and descriptor coverage test look sound, but streaming RPC outcomes can be recorded as successful before the call finishes. Please address the inline finding before merging.

Validation: source review only; tests were not run.

The gateway recorded authentication and authorization outcomes but not the
state changes users and operators made, so there was no structured record of
who created, changed, or deleted a resource or whether the call succeeded.

Every call by a user or mTLS operator to a write-scoped RPC now emits one API
Activity event naming the actor, the method and its create, update, delete,
or other activity, the request ID, and the outcome derived from the gRPC
status. A streamed call is recorded when its response stream ends, with
the status from its trailers. A call or stream dropped before it
finished, such as when its client disconnected, records an unknown
outcome, since the change it started may still apply. An authorization denial raised for the call stands as its only
record. A catalog in the gateway lists the audited RPCs, and a test fails
when a write-scoped RPC is missing from it.

Refs #2911

Signed-off-by: Kris Hicks <khicks@nvidia.com>
@krishicks
krishicks force-pushed the feat/2911-record-gateway-mutations/krishicks branch from c82b1b3 to 66b5ccb Compare November 16, 2026 18:50

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked 66b5ccb. The streaming outcome finding is resolved: the audit wrapper now observes the final gRPC trailer status and records Unknown when a response stream is dropped before completion. No new findings.

Validation: six focused tests passed against the extracted body wrapper, covering trailer success and failure, drops before polling and after data, body errors, empty bodies, and denial suppression. The full server suite was not run.

@krishicks
krishicks added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit e8ba30f Nov 16, 2026
92 of 94 checks passed
@krishicks
krishicks deleted the feat/2911-record-gateway-mutations/krishicks branch November 16, 2026 19:41
cv added a commit to NVIDIA/NemoClaw that referenced this pull request Nov 18, 2026
Moves NemoClaw's OpenShell pins from v0.1.8 to v0.1.9: the native CLI,
the gateway, supervisor and sandbox images, the gateway chart, the
OpenShell crates, and the vendored SDK. A published managed Kubernetes
gateway now disables the chart's metrics with `metrics.enabled: false`,
which v0.1.9 requires, and the live middleware fixture moves to
OpenShell's HTTP hooks. Deployments built from this revision run
OpenShell v0.1.9, whose chart applies `caConfigMap`'s CA only to the
OIDC issuer.

## Problem

NemoClaw pins OpenShell v0.1.8 (`f3d9765`). v0.1.9 (`ce58cf8`, 51
commits later) is the latest release, and it changes three things
NemoClaw depends on:

- The chart refuses `service.metricsPort: 0` (NVIDIA/OpenShell#5478),
which NemoClaw sets on a gateway published through `expose.service`; the
chart's exposed contract scenario fails to render with
`service.metricsPort must be an integer between 1 and 65535; to disable
metrics, set metrics.enabled=false`.
- `server.oidc.caConfigMapName` no longer sets a process-wide
`SSL_CERT_FILE` on the gateway; the chart renders
`[openshell.gateway.oidc] ca_bundle`, so the CA applies only to OIDC
discovery and JWKS requests (NVIDIA/OpenShell#3673).
- Middleware selectors with a port fail validation, and the legacy
`EvaluateHttpRequest` and `HttpResponsePreReturn` hooks are deprecated,
to be removed in 0.2.0 (NVIDIA/OpenShell#4359).

Part of #13829: it settles question 2 of [cv's
answers](#13829 (comment)),
keeping `caConfigMap` as a typed field under
`target.kubernetes.authentication`. With v0.1.8 the chart turned that
ConfigMap into the gateway's process-wide `SSL_CERT_FILE`, so a field
named for the issuer's CA also set the trust of the gateway's other
HTTPS clients; with v0.1.9 it affects only the issuer, which is what a
typed authentication field says. The rest of #13829 (pull Secrets,
Docker and Podman ports, the `openshift` target) doesn't depend on this
upgrade, and the OpenShift UID nulls stay until NVIDIA/OpenShell#5483.

## Change

- **Pins:** the Cargo revision of `openshell-core`, `openshell-sdk` and
`openshell-policy`, and of the vendored SDK's `openshell-core` and
`openshell-crypto`; in `versions.json`, the version, revision, Linux CLI
archives, gateway, supervisor and sandbox images, and gateway chart; the
managed-gateway reference, a contract test and the gateway contract
module, which carry the gateway digest. The CLI checksums come from the
release's `openshell-checksums-sha256.txt`; the image and chart digests
come from GHCR's `0.1.9` tags, and GHCR's `0.1.8` tags match the
previous pins. `Cargo.lock` changes only the five OpenShell crates'
revision.
- **Vendored SDK:** upstream changed `src/client.rs` and `src/raw.rs` (a
raw `ServiceDiscovery` client), `src/transport.rs` and
`tests/client_mock.rs`, copied unchanged. The SDK manifest still enables
core telemetry, so the manifest patch stays; `NOTICE.md` records the
refresh. The SDK integration limits in
`internal-docs/design/architecture.md` still hold at v0.1.9 and cite
that revision.
- **Published gateway metrics:** `expose.service` sets the chart's
`metrics.enabled: false` instead of `service.metricsPort: 0`, so the
NodePort Service still publishes only the gRPC port and the gateway
serves no plaintext metrics. This deletes the `metricsPort` workaround
and its comment, which #14313 left for this upgrade. The chart
contract's exposed scenario failed with the v0.1.9 chart before the
change and passes after it.
- **Middleware:** the `docker-dummy` live case's middleware fixture
implements OpenShell's `HttpHooks` service and requires the
`openshell.supervisor-middleware.http-hooks` capability, deciding at
preflight without the body; it answers the legacy `EvaluateHttpRequest`
with `UNIMPLEMENTED`. The case's registration and assertions are
unchanged. A middleware selector with a port is a new case in the SDK's
OpenShell-validation test, which failed against v0.1.8 and passes
against v0.1.9; the selector's schema description and the sandbox
network guide say hosts take no port.
- **Docs:** OpenShell links move to the v0.1.9 revision; each path and
anchor exists there, and the three proxy line ranges in the sandbox
network guide move 71 lines down to the same text. The Kubernetes guide
says the gateway trusts `caConfigMap`'s CA only for the issuer.
- **Release notes** list the new pin and the v0.1.9 changes users see,
with two breaking changes: a middleware selector with a port fails plan,
and a DNS-glob endpoint no longer admits a private IP literal on its own
(NVIDIA/OpenShell#5522).

## Upgrade notes checked

- **Supervisor shutdown** (NVIDIA/OpenShell#5342): a dedicated
supervisor now stops within 8 s and Docker waits 10 s, so stops no
longer end in `SIGKILL`, and a connection that drops during shutdown is
no longer recovered. That can shorten the 13 s between stop and start in
#13953's failure, but the failure itself, a restarted supervisor finding
its sandbox session terminal, isn't addressed, NVIDIA/OpenShell#4910 is
still open, and #13953 didn't reproduce in 260 runs; #13953 stays open.
- **OpenShift UIDs:** NVIDIA/OpenShell#5483 is still open and not in
v0.1.9, so `openshift` keeps nulling the chart's UIDs (#13828).
- **OCSF** (NVIDIA/OpenShell#5606): NemoClaw reads no OCSF records; the
release notes tell log consumers how to select denials now.
- **Go SDK move, mutation admission limits, startup windows, service
discovery:** NemoClaw uses the Rust SDK and OpenShell's defaults;
authored gateway configuration can set the new keys.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants