Skip to content

test(continuity): upgrade rpm packages on Podman - #4202

Open
elezar wants to merge 5 commits into
mainfrom
codex/rpm-upgrade-tests
Open

elezar wants to merge 5 commits into
mainfrom
codex/rpm-upgrade-tests

Conversation

@elezar

@elezar elezar commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the RPM variant of the upgrade disruption from #3521, and cover Podman in the continuity matrix. A Fedora guest starts on the latest published release's openshell and openshell-gateway packages, upgrades to the candidate, and the continuity scenarios check that a sandbox created before the upgrade still works afterwards. The upgrade runs under rootless and rootful Podman, and gateway-restart runs under rootful.

Stacked on #3521, which holds the continuity family, the runner's --disruption support, the Debian upgrade, and the CI wiring. Its base stays codex/ubuntu-deb-upgrade-test until that merges.

Related Issue

Refs #2975 — reusable N-to-N+1 upgrade qualification.

Changes

  • Add the rpm-upgrade-source installer. It provisions the guest from the latest release's two RPMs and its release-tagged images, and stages the candidate RPMs and images with the upgrade disruption, as deb-upgrade-source does for Debian.
  • Add the RPM upgrade step, rendered by the installer for the user that runs the gateway: the login user under rootless Podman, root under rootful. It loads the candidate's images into that user's Podman storage, installs both packages with dnf, requires both installed versions to equal the candidates', and runs the gateway on the candidate's images.
  • Select the candidate's runtime images before installing, and restart the gateway only if the package did not. The gateway package restarts the user service itself after an upgrade, and a second restart right after it interrupted the new gateway while it brought the sandbox back, leaving the sandbox in an error phase.
  • Teach the prepare-upgrade-source action the rpm package type. It downloads both RPMs and verifies each against the release's openshell-checksums-sha256.txt.
  • Add continuity rows: upgrade on fedora-podman-rootless and fedora-podman-rootful, and gateway-restart on fedora-podman-rootful. Like the other continuity rows, they run in the release workflows. Document them in TESTING.md and CI.md.

Release Notes

Testing

Ran on macOS with HVF on Fedora 44 (arm64) guests, with the baseline v0.1.9 and the dev build as the candidate. These runs predate rebasing onto #3521's current head. The files this PR adds or changes are unchanged by the rebase, but the newer main also changed tmachine's own Rust sources, so the guest runs have not been repeated on this head.

  • continuity with upgrade on rpm-upgrade-source: passes on fedora-podman-rootless and on fedora-podman-rootful. The first rootless run failed because the step restarted the gateway twice, as described above, and the first rootful run failed on a check that returned non-zero for root.
  • continuity with gateway-restart on the rpm installer: passes on both fedora-podman-rootless and fedora-podman-rootful.
  • shellcheck on the rendered step for both users, and on the action's download script; actionlint on e2e-suites.yml; row selection gives 30 rows without upgrade-source and 33 with it, with unique artifact names.

Not verified:

  • The new CI jobs have not run on this head. The rpm branch of prepare-upgrade-source has only been read and linted: it runs in CI on amd64, and the local runs used the arm64 packages.

  • Four local guest runs failed with an SSH Permission denied on a guest tmachine had already reached, at points unrelated to this change. I haven't found the cause. Continuity rows now gate the release workflows, so a recurrence in CI could fail a run for reasons unrelated to OpenShell.

  • The pre-commit hooks were skipped: rust:lint crashes with a clippy internal compiler error in openshell-supervisor-network, which this PR doesn't touch.

  • Checks appropriate to the affected code and behavior pass

  • Unit tests added/updated (if applicable)

  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Oct 5, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch 3 times, most recently from dff9ea0 to cb06c80 Compare October 8, 2026 12:12
@github-actions

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 23, 2026
@SDAChess
SDAChess force-pushed the codex/ubuntu-deb-upgrade-test branch from cb06c80 to 295e946 Compare October 23, 2026 12:28
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Oct 24, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Nov 8, 2026
@purp purp added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 15, 2026
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Nov 15, 2026
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from 295e946 to f2f756f Compare November 19, 2026 14:38
@elezar
elezar force-pushed the codex/rpm-upgrade-tests branch from 148cadd to edb874b Compare November 19, 2026 15:19
@elezar elezar changed the title test(upgrade): qualify RPM package upgrades test(continuity): upgrade rpm packages on rootless Podman Nov 19, 2026
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch 2 times, most recently from f580d45 to d3f14ae Compare November 19, 2026 19:02
@elezar
elezar force-pushed the codex/rpm-upgrade-tests branch from edb874b to c22cd2b Compare November 19, 2026 20:04
@elezar elezar changed the title test(continuity): upgrade rpm packages on rootless Podman test(continuity): upgrade rpm packages on Podman Nov 19, 2026
@elezar
elezar marked this pull request as ready for review November 19, 2026 20:08
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from d3f14ae to 9a30af5 Compare November 19, 2026 20:28
@elezar
elezar force-pushed the codex/rpm-upgrade-tests branch from c22cd2b to f044425 Compare November 19, 2026 20:55

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Note it has had no CI run on this head (stacked on #3521, continuity skipped) and the local runs used arm64 while CI is amd64, so the rpm download path is untested in CI.

The four unexplained SSH Permission denied flakes concern me now that continuity gates releases. Could we agree a plan (retry, or non-blocking until understood) before this merges?

Also, the rpm step waits up to 10s to detect the package's own restart. If there's a way to assert it from the unit's InvocationID instead of polling, I'd prefer that.

@elezar elezar added test:e2e Requires end-to-end coverage test:e2e:continuity Run the continuity suites (gateway restart and upgrade) on PR mirrors labels Nov 20, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for f044425. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Base automatically changed from codex/ubuntu-deb-upgrade-test to main November 20, 2026 09:47
@elezar
elezar dismissed matthewgrossman’s stale review November 20, 2026 09:47

The base branch was changed.

@elezar

elezar commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

The four unexplained SSH Permission denied flakes concern me now that continuity gates releases. Could we agree a plan (retry, or non-blocking until understood) before this merges?

I have a local commit to switch to ussing SSH_ASKPASS instead. An agent discovered that the issue seems to have been caused by sshpass behaving unreliably and sending partial passwords.

@elezar

elezar commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Also, the rpm step waits up to 10s to detect the package's own restart. If there's a way to assert it from the unit's InvocationID instead of polling, I'd prefer that.

I would need to check this.

@elezar

elezar commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Re the 10s wait in the rpm upgrade step: it already asserts from the unit's InvocationID. It records the ID before dnf install and waits for it to change. The wait is there because I haven't confirmed that the package's %systemd_user_postun_with_restart has finished by the time dnf returns. If it hasn't, a single read would report no restart and the script would restart the unit a second time, which is what broke Podman's sandbox recovery earlier. It is capped at 10s and normally exits on the first change. I'd rather measure whether the restart is already visible when dnf returns and, if so, replace the loop with one read and a clear failure message. I'll track that as a follow-up rather than hold this PR for it.

Add an `rpm-upgrade-source` installer that provisions a Fedora guest from the
latest release's openshell and openshell-gateway packages and release-tagged
images, and stages the candidate packages and images with the `upgrade`
disruption that installs them. The scenarios stay unchanged and run under it
like any other disruption.

The upgrade step selects the candidate's runtime images before it installs,
because the gateway package restarts the user service itself after an upgrade
(%systemd_user_postun_with_restart). It restarts the service only if the
package did not: a second restart right after the package's interrupts the new
gateway while it brings its sandboxes back, which left the pre-upgrade sandbox
in an error phase. It checks both installed packages against the candidates'
versions and supports rootless Podman only.

Teach the prepare-upgrade-source action the rpm package type, which downloads
both RPMs and verifies them against the release's single checksum file, and
add a fedora-podman-rootless row that runs with `upgrade-source`.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Render the rpm upgrade step for the user that runs the gateway, which the
packaged-gateway role detects: the login user under rootless Podman and root
under rootful. Every action in the step, including loading the images into
Podman storage, writing gateway.env, and talking to the user manager, runs as
that user. The step no longer refuses rootful Podman, and checks that the
service is loaded for the gateway user through `show` instead of `cat`, which
fails for root.

Add fedora-podman-rootful rows to the continuity matrix: gateway-restart on
the rpm installer, and upgrade on rpm-upgrade-source with the other upgrade
rows.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
sshpass sometimes answered the password prompt with a password sshd rejected, which failed about 3% of logins on a loaded host with 'Permission denied'. tmachine opens a new connection for every command, so a run made hundreds of logins and hit the flake often. Pass the password through SSH_ASKPASS instead. It failed none of 180 logins in the same test.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
The baseline is the latest release when the run starts, so the same candidate can pass one day and fail the next. Write the tag to the run summary and a notice annotation, in the select job and in each upgrade row, so a result says which release it upgraded from.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Record that the wait for the package's restart should be measured and, if the restart is already visible when dnf returns, replaced with a single InvocationID read.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/rpm-upgrade-tests branch from f044425 to dbcc79f Compare November 20, 2026 14:37
@elezar
elezar requested a review from a team as a code owner November 20, 2026 14:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. test:e2e:continuity Run the continuity suites (gateway restart and upgrade) on PR mirrors test:e2e Requires end-to-end coverage topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants