Skip to content

fix(sfcompute): keep starting nodes listed as pending on SSH 404 - #163

Merged
drewmalin merged 1 commit into
mainfrom
BREV-13508/keep-starting-sfcompute-nodes-listed
Nov 13, 2026
Merged

drewmalin merged 1 commit into
mainfrom
BREV-13508/keep-starting-sfcompute-nodes-listed

Conversation

@mmoulikk

@mmoulikk mmoulikk commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Issue

While a new SF Compute node is starting, SF Compute already lists it as running, but GET /integrations/brev/v1/instances/{id}/ssh returns 404 ("SSH details not currently available") for a few seconds. ListInstances left the node out of the list, and dev-plane reads a missing node as terminated. The environment then moved to instance_lost and could end up in terminated: setup never ran, Delete was hidden in the console, and the VM kept running and billing (BREV-13508).

Root cause

getSSHInfo returned every SSH lookup error to its caller. ListInstances skipped any node whose lookup failed, and GetInstance returned the error. A 404 right after creation means the SSH details aren't ready yet, not that the node is gone.

Fix

  • getSSHInfo: a 404 within 45 minutes of creation (sshDetailsPendingWindow) means the node is still starting. The node stays listed as pending, GetInstance no longer errors, and it logs sfcv2: SSH details not available yet, reporting instance as pending.
  • sfcInstanceToBrevInstance: a running node without an SSH hostname is reported as pending, since Brev can't reach it yet.
  • After 45 minutes, and for any other error (429, 5xx, connection), nothing changes and the node is skipped as before. 45 minutes is longer than the build's SSH wait (up to 3 × 10 min), so a node that never gets SSH details still fails its build.
  • pending doesn't change an environment's state in dev-plane, and time in pending isn't billed.

Test

  • Unit tests in instance_test.go, against a fake SF Compute API:
    • TestIsSSHDetailsPending: a 404 inside the window counts as pending; a 404 after the window, a 500, or a connection error doesn't.
    • TestListInstancesKeepsNewInstanceWithoutSSHDetailsAsPending: a new node with an SSH 404 is listed as pending, a ready node is running with its SSH address, and an old node with an SSH 404 is still left out.
    • TestGetInstanceReportsNewInstanceWithoutSSHDetailsAsPending: GetInstance returns pending instead of an error.
  • CI passes.

Staging (dev-plane v0-brev-13508-0, Nov 13)

  • Created 11 SF Compute environments. 8 nodes hit the SSH 404 and were kept as pending instead of dropped (logs).
  • No node was dropped after the deploy. The only "skipping instance due to SSH error" came from the previous build, at 11:05 UTC (log).
  • No environment went to instance_lost.
  • dau23x56n (logs) and coyxnym6e (logs) hit the 404, stayed in building, and reached running after one build retry while SSH came up. The other 9 were deleted on purpose mid-build, as part of the test.

@mmoulikk mmoulikk self-assigned this Nov 4, 2026
@mmoulikk
mmoulikk force-pushed the BREV-13508/keep-starting-sfcompute-nodes-listed branch from 5247e92 to 39d0ee2 Compare November 13, 2026 05:51
@mmoulikk
mmoulikk marked this pull request as ready for review November 13, 2026 12:43
@mmoulikk
mmoulikk requested a review from a team as a code owner November 13, 2026 12:43
@drewmalin
drewmalin merged commit 303c6ee into main Nov 13, 2026
5 checks passed
@drewmalin
drewmalin deleted the BREV-13508/keep-starting-sfcompute-nodes-listed branch November 13, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants