Repository navigation
feat(testkit): run conformance as one program and add unsupported-test entries - #5755
Conversation
…t entries Make the openshell-conformance bin the conformance test target (harness = false) so run, cargo test and nextest execute the same program. Add OPENSHELL_TEST_UNSUPPORTED, a list of <test>=<issue url> entries that every interface reports as ignored, with startup, summary and JUnit reporting, strict checks, and a refusal on forced runs. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…der libtest Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…orwarding Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Label |
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…he YAML file Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Label |
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ironment names Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
elezar
left a comment
There was a problem hiding this comment.
Thanks @matthewgrossman.
I think in the long term, we probably want to replace the shell-scripts that we have here with something implemented in rust, but we can handle that as a follow-up instead of blocking on this. It's more important to get the tests migrated than to get this perfect first time.
Summary
Make the
openshell-conformanceprogram the conformance test target, sorun,cargo testand nextest all run the same code. AddOPENSHELL_TEST_UNSUPPORTEDso an environment can mark a test unsupported, with the same skip under every way of running.Related Issue
Closes #5754 (part of #5281)
Why an environment variable
The list is an environment variable and not a
--skipflag:--skipalready means something else in two of the three ways of running tests. libtest's--skip PATTERNleaves out tests whose names match, andcargo nextest runaccepts the same flag after--. Soopenshell-conformance --skip cli::x=https://...orcargo test -- --skip ...would be read as a name filter. It matches no test, so the test runs anyway, with no error. That is a silent misread, not just a confusing name.--list, then<name> --exactfor each test). An environment variable is the only input every runner passes to every test process.openshell-conformance run,cargo testand nextest, which is the point of running one program everywhere.run --unsupportedshortcut would be safe, because libtest rejects flags it doesn't know, so it could not be misread the way--skipwould. Leaving it out keeps exactly one way to give the list.Known limitation
Scoping by crate is what lets one variable serve several programs, but it means a misspelled crate prefix (
conformence::smoke::x=...) is silently ignored. A program alone can't tell a typo from an entry meant for another program.unsupported.yamlis validated in Branch Checks (test names and issues through the program, environment names against the rows and host lanes).CI wiring
Unsupported tests are listed in
tests/suites/conformance/unsupported.yaml. CI turns the entries for an environment intoOPENSHELL_TEST_UNSUPPORTED, so the program interface stays the one variable. A migration PR that hits an environment-specific bug files an issue and adds an entry; the fixing PR deletes it.tasks/scripts/conformance-unsupported.sh <environment>is oneyqcall that prints that environment's entries as<test>=<issue>,…. Branch Checks runsopenshell-conformance listwith each environment's entries, so the program's own checks (unknown test,https://issue, duplicates) validate the file, and it rejects environment names that aren't a conformance row ofe2e-suites.ymlor a host lane.suitesjob sets the variable for<environment>/<installer>, andopenshell-test-runnerpasses it into the guest for bothreadyandrun.e2e/support/conformance.shsets it for the lane's name unless the caller already did.The file is empty. The first entry is likely the on-failure restart test (#5507, a VM driver bug that hangs, so a skip fits), once it is ported to conformance.
The forwarding was checked in CI with the earlier per-row and per-lane inputs: a temporary entry showed
SKIPwith the issue in both the tmachine results and the Docker lane, while other rows still ran the test. The YAML-file sourcing was then checked the same way in a second run, which showed the same skip on theubuntu-docker-rootful/debrow and the Docker lane while the Podman row still ran every test; both temporary entries were reverted.Changes
harness = falseon theopenshell-conformancebin,test = falseon the conformance library. nextest's binary ID changes fromconformancetoconformance::bin/openshell-conformance, and editor per-test "Run" buttons go away. The macro's#[tokio::test]wrappers are unchanged.OPENSHELL_TEST_UNSUPPORTED:<test>=<https issue url>entries separated by commas and/or whitespace. Entries are qualified by crate, and a program applies only entries for crates it links.#[ignore]d test, non-https://issue, malformed entry.listmarks them,runprints the active entries once and lists them in its summary; undercargo testand nextest the program stays quiet and the caller lists the entries, and its JUnit marks themskippedwith anunsupportedproperty.runpasses the variable to its test processes.unsupported: <issue>; remove the entry from OPENSHELL_TEST_UNSUPPORTED to run it.Testing
Also checked by hand on conformance without a gateway:
cargo test,cargo nextest runandopenshell-conformance runshow the same skip for an entry, a forced run fails under each, and a bad entry exits 2 (nextest reports its own list-failure code, 104, which its listing step uses for any failure).Checklist