Skip to content

test(testkit): support per-environment skips in all suites - #5789

Open
matthewgrossman wants to merge 4 commits into
mainfrom
test/5788-unsupported-features-drivers/matthewgrossman
Open

matthewgrossman wants to merge 4 commits into
mainfrom
test/5788-unsupported-features-drivers/matthewgrossman

Conversation

@matthewgrossman

Copy link
Copy Markdown
Member

Summary

Feature and driver suites now honor OPENSHELL_TEST_UNSUPPORTED, so tests can stay in CI rows even when a specific environment can't pass them yet.

Related Issue

Closes #5788

Changes

  • Run all nine feature and driver integration targets through testkit::main(), with unique target names and unchanged nextest package selection. Keep the macro's cfg(test) wrapper for in-flight ports; removing it stays with test(testkit): run every testkit suite as one program, and let nextest run the prebuilt program #5765.
  • Share tests/suites/unsupported.yaml and its reader across all families. Log issue URLs on nextest rows and document how to add entries in the testkit README.
  • Extend Branch Checks to enforce harness settings and validate the list against all suite workspaces and CI environments.

For new feature or driver suites, copy these five lines: the first four go in Cargo.toml (choose a globally unique target name), and rustfmt expands the last line in the test file.

[[test]]
name = "my-feature"
path = "tests/my_feature.rs"
harness = false
fn main() -> std::process::ExitCode { testkit::main() }

Testing

  • cargo fmt --check and Clippy (--all-targets -- -D warnings) for features, Podman and conformance; ShellCheck and actionlint for the changed scripts and workflow.
  • Nextest discovery unchanged: 37 features tests, two Podman tests and 49 conformance tests. All 14 CDI unit tests passed. An unskipped CDI integration test reached gateway setup through testkit.
  • check-conformance-program.sh passed with the checked-in list and with temporary entries for all three families. Verified CDI shows SKIP and conformance lists the issue URL. Negative checks rejected an unknown crate, environment and test, an invalid URL, duplicate entries and a missing harness setting.
  • Full tmachine E2E rows not run locally.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 20, 2026
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Allow nextest to run zero tests when the environment has unsupported entries. This also allows an empty filter match in that case, but avoids a second archive extraction and test discovery.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
matthewgrossman marked this pull request as ready for review November 20, 2026 05:28
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 20, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 91fc8b7

@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 91fc8b7. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(testkit): let features and drivers suites read unsupported.yaml

1 participant