Skip to content

test(conformance): run the conformance suite against a gateway that pushes configuration - #5622

Open
pimlock wants to merge 9 commits into
mainfrom
test/conformance-config-push/pimlock
Open

pimlock wants to merge 9 commits into
mainfrom
test/conformance-config-push/pimlock

Conversation

@pimlock

@pimlock pimlock commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Runs the full conformance suite in CI against a gateway with config_delivery_mode = "push", ahead of making push the default in #4498.

Related Issue

Refs #4498. No issue required: test and CI coverage only, no product behavior change.

Changes

  • New tmachine feature environment ubuntu-docker-rootful-config-push: ubuntu-docker-rootful plus playbooks/config-push.yaml, which writes config_delivery_mode = "push" to /etc/openshell-test/gateway-overrides.toml. The deb, rpm and binaries installers already merge that file.
  • One row in .github/workflows/e2e-suites.yml: conformance in the new environment with the deb installer.
  • mise run e2e:testkit:push runs conformance against a local Docker gateway with OPENSHELL_CONFIG_DELIVERY_MODE=push, for hosts without KVM.

Nothing asserts push is in use. A typo in the setting fails gateway startup (unknown fields are rejected), and supervisors built from this tree always support push, so push is off only if the override file stops being merged. A separate evidence suite was tried and dropped as not worth a VM per E2E run.

Scope notes:

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Local results, before merging main:

  • tmachine (aarch64 host), ubuntu-docker-rootful-config-push + deb: conformance 22/22 pass.
  • Local Docker gateway in push mode: conformance 22/22 pass; the gateway log shows all 27 supervisor sessions accepted with config_push=true.

The new CI row runs on amd64 and has not run in CI yet.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…ushes configuration

Add the ubuntu-docker-rootful-config-push tmachine environment. Its setup
playbook, config-delivery.yaml, writes config_delivery_mode to the gateway
overrides that the deb, rpm and binaries installers apply, and publishes the
mode as OPENSHELL_TEST_CONFIG_DELIVERY_MODE.

A push gateway lets a supervisor without push support keep polling, so the
conformance suite passes in either mode. The new config-push feature suite
fails unless the published mode is push and a new sandbox's supervisor
reports a pushed-configuration session.

E2E Suites runs the conformance family and the config-push suite in the new
environment with the deb installer. mise run e2e:testkit:push runs both
against a local Docker gateway in push mode.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
The supervisor logs a pushed-configuration session only when the gateway
accepted it with push enabled, so the config-push suite no longer also
requires an OPENSHELL_TEST_CONFIG_DELIVERY_MODE variable. The environment's
playbook, now config-push.yaml, sets push without a mode input, since
polling is going away, and no longer writes an [openshell] table that a
second overrides writer could duplicate.

The test retries a failed `openshell logs` call until its timeout.
e2e:testkit:push runs the push check before the conformance suite, so a
conformance failure can't hide whether push was on. The TESTING.md section
moves next to the tmachine runner docs instead of capturing the feature
suite paragraphs that follow the gateway overrides section.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 16, 2026
…teway-push-1d7bb5

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

# Conflicts:
#	.github/workflows/e2e-suites.yml
#	tests/suites/features/Cargo.toml
#	tests/tmachine/config.nix
@github-actions

Copy link
Copy Markdown

@pimlock
pimlock marked this pull request as draft November 16, 2026 21:48
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
A gateway built from this tree rejects unknown config fields and its
supervisor always supports push, so the push conformance row already
runs with push on unless the override file stops being merged. The
separate suite cost a VM per E2E run for that one case.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
… file

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
pimlock marked this pull request as ready for review November 16, 2026 23:54
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 002bf18. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This merges cleanly into current main, but I’d rebase once #5780 lands so the push lane uses the shared gateway override role. The amd64 push row already passed on this PR head, so the PR description can drop the note that it hasn’t run in CI. The next run should verify push after the rebase.

# as a gateway.d drop-in config, which must declare `[openshell]
# version = 2`. When it lands, add that table here unless an earlier
# writer already does; TOML rejects a table defined twice.
- name: Write the configuration delivery mode

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #5780 lands, please switch this playbook to the shared gateway_overrides role with the config-push block name. That lets you remove the duplicate file-writing tasks and the warning about the other writers replacing the file. It also keeps the future gateway drop-in change in one place.

{"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"conformance","gateway":"tmachine","runner":"ubuntu-24.04"},
{"environment":"ubuntu-docker-rootful-config-push","installer":"deb","family":"conformance","package":"conformance","gateway":"openshell","runner":"ubuntu-24.04"},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rerun ubuntu-docker-rootful-config-push/deb after the rebase and confirm from the gateway log that a supervisor session actually negotiated push. If the installer drops the override, conformance can still pass in poll mode. Keep this environment name stable for #5792’s config_push and policy_activation ports; the ported push-session check can provide the lasting assertion. The row will pick up unsupported.yaml from main without extra wiring; only add a row-specific skip for a reproduced failure with a tracking issue.

Comment thread tasks/test.toml
["e2e:testkit:push"]
description = "Run the conformance suite against a Docker gateway that pushes configuration"
depends = ["e2e:workload:build"]
env = { OPENSHELL_CONFIG_DELIVERY_MODE = "push" }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reject a nonempty OPENSHELL_GATEWAY_ENDPOINT here. The wrapper reuses that gateway instead of starting one, so setting this variable to push can silently test a polling gateway. Also consider using the shared Docker conformance helper so this local task picks up the Docker entries from unsupported.yaml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants