Deferred: re-test harden-runner v2.22.3 in block mode (phase 5) #24

Why

Block mode for step-security/harden-runner passed the V10 egress run, but 2 of 24 block-mode jobs hung about 15 minutes in "Complete job" until GitHub cancelled them. Both hung jobs ran in westus2. They were the only westus2 jobs of the 22 that ran. The 20 jobs in other regions did not hang. The gate failed closed. See V10 item 10 and K-14 for the full record.

Upstream issue 708 reports the same symptom: the agent drops Azure DNS 168.63.129.16 and the job hangs in cleanup.

Current state

• Every job runs harden-runner in audit mode, pinned to v2.22.2, with a measured allowlist (K-14).
• Issue 708 is open. v2.22.3 (46b06cea07f9e64621fdf066ba77ac6915163d43, released 2026-10-27) is the candidate fix.
• Phase 5 of the build plan holds the block-mode revisit. Owner deferred the test on 2026-10-31.

New risk: the feature-flag fetch

In agent v0.18.0 the DNS revert runs only if StepSecurity's feature-flag API returns revert_github_dns_experiment: true.

• The fetch is one request with a 3 s timeout. No retry. No backoff.
• If it fails, the flags default to false and RevertGithubDNSExperiment is skipped. The old hang can return.
• The fetch happens before DNS setup. No action input forces the revert.
• Code: agent.go:70 and :85-87 and global_feature_flags.go:35-49.
• On 2026-10-30 the maintainer traced the v2.22.3 failure in issue 708 to a transient us-west-2 API Gateway timeout. The 30-minute "Cleaning up orphan processes" hang in that report is not addressed.

So the fix depends on StepSecurity API availability. This is an availability risk (the run fails closed and passes on re-run). It is not a security risk.

Test plan

 audit measure --> soak (block, 24-job matrix) --> central runs (block) --> release PR

1. Audit measurement: run v2.22.3 in audit mode. Measure the host lists for test.yml and eval.yml (not measured in V10). Check the central lists again.
2. Soak workflow: put hr-soak.yml in the scratch repo claude-review-p0-source. It has a 24-job matrix that copies the central job steps. It uses block mode, disable-telemetry: true, no secrets and no Claude calls. Dispatch it at least 5 times over at least 2 days. Record the region, the feature-flag value and the last log line of each job.
3. Central runs: run the full review workflow in block mode on 3 test PRs in claude-review-test. Use the test ruleset and the scratch federation rule (the Anthropic rule used only for tests). The owner points the rule at the candidate commit with make release-rule-test.

Pass rule

• At least 71 block-mode jobs with 0 hangs.
• At least 10 of them in westus2.
• No step fails on a dropped allowlisted IP.
• Clean test PRs pass the gate. The vulnerable test PR fails only through dependency review.

Why 71: with 0 hangs in n jobs, the 95% upper bound on the hang rate p is where (1 - p)^n = 0.05. For p = 1 in 24 (the V10 rate was 2 of 24), n = 71. Why 10 in westus2: the old failure was near 100% there, and 10 clean jobs give p < 0.01 against "not fixed". GitHub-hosted runners do not allow region choice, so add dispatches until 10 westus2 jobs exist. Stop at 240 jobs and report.

Inconclusive: if no westus2 job logs that GitHub's DNS experiment was detected, the test does not prove the fix. Count each job with RevertGithubDNSExperiment:false. If one hangs, it counts as a hang.

Owner steps

1. Approve the test scope: the soak workflow, 3 never-merged branches, the ruleset pin moves (always restored) and the rule move.
2. Approve 1 eval environment run (audit measurement).
3. Run make release-rule-test RULE=<scratch rule> SHA=<candidate> (Console browser login).
4. Read the result. Decide: pass, or inconclusive (switch with the risk recorded, or wait).
5. Release: approve the release eval, then run make release SHA=<merge SHA>.

Rollback

• During the test: pin the test ruleset back to the current release SHA. The scratch rule can stay.
• After release: make release SHA=<previous release SHA>, or merge a PR that sets audit mode and release that.
• If any hang occurs: stay in audit mode, record job IDs, region and agent log lines in the spec, and consider reporting on issue 708.

Cost

• No Actions minutes billed (public repos, standard runners). About 135 to 250 short jobs.
• Claude calls on the test workspace only: the central runs and 1 eval run. The soak makes none. The owner reads the spend in the Console.
• Time: about 1 hour for the audit measurement, at least 2 days for the soak (spread on purpose), about 1 hour for the central runs.

Open questions

• Pin-age rule: the owner allows an exception to the 7-day rule for v2.22.3 and is considering a change from 7 days to 2 days. Decide, then record it in the build plan.
• Newer release: if v2.22.4 or later appears first, either stay on tested v2.22.3 or re-run the soak on the new release before the pin.