Skip to content

fix(sandbox): replace lifetime exec cap with retry deadlines - #4105

Merged
drew merged 3 commits into
mainfrom
codex/4103-expire-exec-requests/drew
Oct 2, 2026
Merged

drew merged 3 commits into
mainfrom
codex/4103-expire-exec-requests/drew

Conversation

@drew

@drew drew commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Long-lived sandboxes stop accepting new execs after 4,096 request IDs. Replace that lifetime cap with a 30-second admission and recovery deadline so fresh execs can continue while expired requests fail without running a command twice.

Related Issue

Closes #4103

Changes

  • Carry an absolute exec expiration time in the request envelope and include it in the payload digest. Transport retries preserve the same ID, payload, and deadline.
  • Retain request IDs until expiration, then reclaim them as new exec requests arrive. Reject expired requests before starting or reattaching a process, and prevent clock rollback from resurrecting discarded IDs.
  • Preserve existing process retention limits and allow admitted commands to run beyond the recovery deadline. Report an uncertain execution outcome when startup recovery expires.
  • Update the crate protocol notes. This private protocol change requires updating the supervisor and sandbox runtime together; their wall clocks must be synchronized.

Testing

  • Checks appropriate to the affected code and behavior pass: cargo clippy -p openshell-sandbox-backend -p openshell-sandbox --all-targets -- -D warnings, Rust formatting, and git diff --check.
  • Unit tests added/updated: 233 tests pass with cargo test -p openshell-sandbox-backend -p openshell-sandbox --lib. Coverage includes 10,000 admitted IDs, expiry and reclamation, delayed first attempts, missing deadlines, clock rollback, and digest-bound deadlines. The real boundary process test also launches and recovers an exec after 4,096 earlier IDs.
  • E2E tests added/updated: four Docker exec tests pass using supervisor and sandbox images built from this checkout. They cover large output, piped stdin, environment propagation, and a 31-second command followed by a fresh exec in the same sandbox.
  • mise run docs passes with no errors and three warnings.
  • The repository pre-commit hook (mise run pre-commit) passes, including workspace Clippy, formatting, license, Markdown, Helm, protobuf, Python, and TypeScript checks.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated: private exec protocol and deployment compatibility documented in the crate README.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
drew requested review from mrunalp and sjenning as code owners October 2, 2026 04:39
@drew drew added the area:sandbox Sandbox runtime and isolation work label Oct 2, 2026
@drew
drew requested review from a team and derekwaynecarr as code owners October 2, 2026 04:39
@drew drew added the area:sandbox Sandbox runtime and isolation work label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

drew added 2 commits October 2, 2026 08:53
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>

@purp purp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚢

@drew
drew added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 121930b Oct 2, 2026
93 checks passed
@drew
drew deleted the codex/4103-expire-exec-requests/drew branch October 2, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox runtime and isolation work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(sandbox): long-lived sandboxes stop accepting execs after 4096 requests

3 participants