Skip to content

feat(testkit): run conformance as one program and add unsupported-test entries - #5755

Merged
matthewgrossman merged 13 commits into
mainfrom
feat/testkit-unsupported-tests/matthewgrossman
Nov 19, 2026
Merged

matthewgrossman merged 13 commits into
mainfrom
feat/testkit-unsupported-tests/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Make the openshell-conformance program the conformance test target, so run, cargo test and nextest all run the same code. Add OPENSHELL_TEST_UNSUPPORTED so an environment can mark a test unsupported, with the same skip under every way of running.

Related Issue

Closes #5754 (part of #5281)

Why an environment variable

The list is an environment variable and not a --skip flag:

  • --skip already means something else in two of the three ways of running tests. libtest's --skip PATTERN leaves out tests whose names match, and cargo nextest run accepts the same flag after --. So openshell-conformance --skip cli::x=https://... or cargo test -- --skip ... would be read as a name filter. It matches no test, so the test runs anyway, with no error. That is a silent misread, not just a confusing name.
  • A flag can't reach the program under nextest. nextest writes the program's command line itself (--list, then <name> --exact for each test). An environment variable is the only input every runner passes to every test process.
  • So one uniquely named variable is the one interface that behaves the same under openshell-conformance run, cargo test and nextest, which is the point of running one program everywhere.
  • "Unsupported" describes the environment ("this doesn't work here yet, see the issue"), not an action like a filter.
  • A later run --unsupported shortcut would be safe, because libtest rejects flags it doesn't know, so it could not be misread the way --skip would. Leaving it out keeps exactly one way to give the list.

Known limitation

Scoping by crate is what lets one variable serve several programs, but it means a misspelled crate prefix (conformence::smoke::x=...) is silently ignored. A program alone can't tell a typo from an entry meant for another program. unsupported.yaml is validated in Branch Checks (test names and issues through the program, environment names against the rows and host lanes).

CI wiring

Unsupported tests are listed in tests/suites/conformance/unsupported.yaml. CI turns the entries for an environment into OPENSHELL_TEST_UNSUPPORTED, so the program interface stays the one variable. A migration PR that hits an environment-specific bug files an issue and adds an entry; the fixing PR deletes it.

- test: conformance::lifecycle::restart_on_failure
  issue: https://github.com/NVIDIA/OpenShell/issues/5507
  environments:
    - vm                  # a host lane: docker, podman, kubernetes or vm
    - ubuntu-k3s/k3s-ha   # a tmachine row: <environment>/<installer>
  • tasks/scripts/conformance-unsupported.sh <environment> is one yq call that prints that environment's entries as <test>=<issue>,…. Branch Checks runs openshell-conformance list with each environment's entries, so the program's own checks (unknown test, https:// issue, duplicates) validate the file, and it rejects environment names that aren't a conformance row of e2e-suites.yml or a host lane.
  • tmachine rows: the suites job sets the variable for <environment>/<installer>, and openshell-test-runner passes it into the guest for both ready and run.
  • Host lanes (Docker, Podman, Kubernetes, VM): e2e/support/conformance.sh sets it for the lane's name unless the caller already did.

The file is empty. The first entry is likely the on-failure restart test (#5507, a VM driver bug that hangs, so a skip fits), once it is ported to conformance.

The forwarding was checked in CI with the earlier per-row and per-lane inputs: a temporary entry showed SKIP with the issue in both the tmachine results and the Docker lane, while other rows still ran the test. The YAML-file sourcing was then checked the same way in a second run, which showed the same skip on the ubuntu-docker-rootful/deb row and the Docker lane while the Podman row still ran every test; both temporary entries were reverted.

Changes

  • harness = false on the openshell-conformance bin, test = false on the conformance library. nextest's binary ID changes from conformance to conformance::bin/openshell-conformance, and editor per-test "Run" buttons go away. The macro's #[tokio::test] wrappers are unchanged.
  • OPENSHELL_TEST_UNSUPPORTED: <test>=<https issue url> entries separated by commas and/or whitespace. Entries are qualified by crate, and a program applies only entries for crates it links.
  • Strict checks exit 2 before anything is listed or run: unknown, duplicate or #[ignore]d test, non-https:// issue, malformed entry.
  • Unsupported tests are ignored in libtest listing and runs, so every runner shows SKIP. list marks them, run prints the active entries once and lists them in its summary; under cargo test and nextest the program stays quiet and the caller lists the entries, and its JUnit marks them skipped with an unsupported property. run passes the variable to its test processes.
  • Forcing an unsupported test to run fails with unsupported: <issue>; remove the entry from OPENSHELL_TEST_UNSUPPORTED to run it.
  • Both READMEs document the variable.

Testing

  • Checks appropriate to the affected code and behavior pass (testkit and conformance fmt and clippy; testkit tests)
  • Unit tests added/updated (parsing, scoping, each check, JUnit, and program-level checks for each interface)
  • E2E tests added/updated (if applicable)

Also checked by hand on conformance without a gateway: cargo test, cargo nextest run and openshell-conformance run show the same skip for an entry, a forced run fails under each, and a bad entry exits 2 (nextest reports its own list-failure code, 104, which its listing step uses for any failure).

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…t entries

Make the openshell-conformance bin the conformance test target
(harness = false) so run, cargo test and nextest execute the same
program. Add OPENSHELL_TEST_UNSUPPORTED, a list of <test>=<issue url>
entries that every interface reports as ignored, with startup,
summary and JUnit reporting, strict checks, and a refusal on forced
runs.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…der libtest

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…orwarding

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 14478a3. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman removed the test:e2e Requires end-to-end coverage label Nov 19, 2026
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…he YAML file

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for e18761a. Open Branch E2E Checks, find the run for commit e18761a, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman removed the test:e2e Requires end-to-end coverage label Nov 19, 2026
…ironment names

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

@elezar elezar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @matthewgrossman.

I think in the long term, we probably want to replace the shell-scripts that we have here with something implemented in rust, but we can handle that as a follow-up instead of blocking on this. It's more important to get the tests migrated than to get this perfect first time.

@matthewgrossman
matthewgrossman added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 6c3d773 Nov 19, 2026
92 of 93 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(testkit): run conformance as one program and skip unsupported tests per environment

2 participants