Skip to content

test(workloads): add workloads suite with a make workload - #5768

Open
matthewgrossman wants to merge 9 commits into
mainfrom
test/5726-workload-suites/matthewgrossman
Open

matthewgrossman wants to merge 9 commits into
mainfrom
test/5726-workload-suites/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Add the workloads testkit family (tests/suites/workloads, program openshell-workloads), where real tools do realistic tasks in sandboxes. The first workload builds a small C project with make. Each workload has its own image, built during setup of a new tmachine ubuntu-docker-rootful-workloads environment, so other rows don't pay for it. The suite runs with the test:e2e:workloads label (together with test:e2e) and on manual runs.

The make test fails today because of a real bug: make 4.3 makes a no-op setresuid call before each recipe, and the sandbox's seccomp filter denies it (#4102; the fix in #4104 has stalled). The test stays active, so the optional workloads row is red until #4102 is fixed.

Related Issue

Part of #5726
Part of #5281

Changes

  • tests/suites/workloads: new workspace with make::make_builds_a_c_program: make builds the project, the program prints the expected output, and make -q finds nothing to rebuild.
  • tests/suites/workloads/make/Dockerfile: pinned nvcr.io/nvidia/base/ubuntu:24.04, gcc, make, the sandbox user, and the C project in /sandbox/project. The test reads the image name from OPENSHELL_TEST_WORKLOAD_MAKE_IMAGE.
  • tmachine: a workloads feature on the Docker base. Its setup playbook builds the image and publishes the variable. The Dockerfile is a setup input, so changing it rebuilds the cached layer.
  • CI: test:e2e:workloads adds a workloads-integration job, wired like test:e2e:continuity (test(continuity): add suites for gateway restart and package upgrade #3521): it needs test:e2e for the integration inputs and is not a required gate.
  • Registered the workspace for fmt, clippy, and lockfile alignment; documented it in TESTING.md and CI.md.

Testing

  • Clippy, fmt, lockfile alignment, and rust:lockfiles:check
  • actionlint on the changed workflows; ansible-playbook --syntax-check; tmachine layer-key ubuntu-docker-rootful-workloads resolves
  • Ran the suite against a local Docker gateway: the image works and make --version passes; the build fails with make: cc: Operation not permitted (bug(sandbox): GNU Make cannot launch recipes with unchanged UID/GID #4102)
  • ubuntu-docker-rootful-workloads row on CI
  • E2E tests added/updated

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026
@matthewgrossman
matthewgrossman marked this pull request as ready for review November 19, 2026 19:44
Comment thread CI.md Outdated
- `test:e2e-kubernetes` runs Kubernetes E2E with the HA Helm overlay
(`replicaCount: 2` and bundled PostgreSQL) and the credential-driver suite
(Kubernetes Secrets plus Vault) in `Branch E2E Checks`
- `test:e2e-workloads` runs the workloads suite (`tests/suites/workloads`) on

@elezar elezar Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for continuity I've used test:e2e:continuity so what about test:e2e:workloads?

Alternatively we could use a / as in: test:e2e/workloads?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like test:e2e:continuity ¯_(ツ)_/¯

@elezar elezar added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 19, 2026
Comment thread tests/suites/workloads/src/make.rs Outdated
Comment on lines +38 to +40
for (name, contents) in SOURCES {
fs::write(project.join(name), contents).context(format!("write {name}"))?;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we just build this into the workload image?

…ites/matthewgrossman

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage test:e2e:workloads Run the workloads suite on PR mirrors (apply with test:e2e) labels Nov 20, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 7e7441c

@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 7e7441c. Open Branch E2E Checks, find the run for commit 7e7441c, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. state:accepted A maintainer decided OpenShell should pursue this issue test:e2e:workloads Run the workloads suite on PR mirrors (apply with test:e2e) test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants