Skip to content

ci(release): qualify install.sh installs and upgrades, smoke-test published releases, then retire Release Canary #5740

Open

Description

@matthewgrossman

User Story

I maintain OpenShell releases. Users install OpenShell with curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh. I need Release Qualification to run that installer against every pre-release, both as a fresh install and as an upgrade from the previous stable, so a broken install path blocks promotion to stable.

Description

Release Qualification installs candidate .deb and .rpm files directly through the package manager (#3454). The upgrade work in #2975 (#3521, #4202) also uses the package manager. Neither runs install.sh.

The only end-to-end install.sh check is Release Canary. It:

  • installs the rolling dev build, not the pre-release or stable;
  • runs after Release Dev publishes, so it cannot block anything;
  • notifies no one. Its snap lane has been red since 2026-10-29 while v0.1.7, v0.1.8, and v0.1.9 shipped.

install.sh regressions have reached users before; for example, #4412 was an install.sh/brew upgrade failure.

The canary also covers more than a basic install. Removing it must not lose:

  • the Helm chart published to GHCR, installed on kind with the CLI connected to it;
  • snap behavior: install.sh uses system Docker and does not install the Docker snap, the docker interface is connected, the user gateway is enabled only for the installing user and not for a second user (fix(snap): only enable gateway user service for one user #4615), the system gateway stays off, mTLS user authentication is on, and openshell-prover works;
  • install.sh refusing to install on snap hosts with no Docker or with only the Docker snap;
  • installing from the published channels (GitHub release assets, Snap Store, GHCR), which catches broken publishing that artifact-based qualification cannot.

The fix is to add install.sh scenarios to Release Qualification, add a post-publish smoke test, and remove Release Canary once both are required.

Context

Definition of Done

  • Release Qualification runs install.sh from the candidate commit with OPENSHELL_VERSION=<pre-release tag> on a clean host, then creates, executes in, and deletes a sandbox.
  • Release Qualification runs an upgrade scenario: install.sh installs the previous stable, creates a gateway and sandbox, then install.sh upgrades to the pre-release. It verifies that existing state survived and that a new sandbox works.
  • Both scenarios cover deb (Ubuntu), rpm (Fedora), snap (Ubuntu), Homebrew (macOS), and the Helm chart (kind). Helm upgrades use helm upgrade from the previous stable chart, and the CLI comes from install.sh.
  • Homebrew on GitHub-hosted macOS runners checks openshell status only, because the runners do not expose Hypervisor.framework. Sandbox coverage on macOS needs a runner with virtualization or is explicitly documented as not covered.
  • The snap scenarios keep the canary's snap checks: no Docker snap installed, docker interface connected, user gateway enabled for the installing user only (not a second user), system gateway disabled, mTLS user authentication enabled, and an openshell-prover check run.
  • Qualification checks that install.sh fails with a clear error on a snap host with no Docker and on a host with only the Docker snap.
  • deb and rpm scenarios also run on arm64.
  • Both scenarios are required for qualification, so a failure blocks stable promotion.
  • Failures keep installer output and gateway service logs.
  • Release (release.yml) ends with a smoke test that installs the published vX.Y.Z through install.sh from the real channels (GitHub release, Snap Store latest/stable, Homebrew, GHCR chart) and creates a sandbox. Stable artifacts are rebuilt after qualification (chore(release): evaluate promoting qualified pre-release artifacts instead of rebuilding for stable #3632 closed as not planned), so this is the only check on what users download. A failure fails the workflow and notifies the release channel.
  • Release Canary (.github/workflows/release-canary.yml) is removed, and the test-release-canary and release-captain skills are updated.

+++ Agent notes

  • Canary snap lane root cause, if it needs to stay green until removal: GitHub runner provisioning writes XDG_RUNTIME_DIR=/run/user/1001 to /etc/environment (XDG_RUNTIME_DIR is set system-wide in /etc/environment, breaking sessions of other users actions/runner-images#14649). The canary's second user inherits it, and on systemd 257+ (Ubuntu 26.04) snapd's per-user systemctl --user daemon-reload fails. Workaround: sudo sed -i '/^XDG_RUNTIME_DIR=/d' /etc/environment before creating the user.
  • Existing canary jobs to port, by line in .github/workflows/release-canary.yml: macOS Homebrew L22, Ubuntu DEB L56, Fedora RPM L94 (systemd container with Podman), snap with system Docker L194, snap Docker preflight L316, Kubernetes Helm on kind L378 (installs the Agent Sandbox controller via e2e/support/install-agent-sandbox.sh).
  • Local repro observations (arm64 containers, v0.1.9): install.sh uses curl -fLs --retry 3, which hides errors and does not retry mid-transfer resets. The Docker driver pulls the supervisor image before the gateway listens (ensure_runtime_image(..., "supervisor") in crates/openshell-driver-docker/src/lib.rs), so a slow pull exceeds the installer's 30s listener wait. Track separately if confirmed.

+++

Activity

  1. added
    topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
    topic:operabilityInstalling/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance.
    on Nov 19, 2026
  2. changed the title [-]ci(qualification): qualify install.sh fresh installs and upgrades, then retire Release Canary[/-] [+]ci(release): qualify install.sh installs and upgrades, smoke-test published releases, then retire Release Canary[/+] on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic:operabilityInstalling/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions