Skip to content

fix(exec): do not replay a remote command that may have started - #535

Open
pcbeingused333 wants to merge 3 commits into
brevdev:mainfrom
pcbeingused333:fix/exec-do-not-replay-running
Open

pcbeingused333 wants to merge 3 commits into
brevdev:mainfrom
pcbeingused333:fix/exec-do-not-replay-running

Conversation

@pcbeingused333

Copy link
Copy Markdown

Fixes #457

brev exec treated every error from the first SSH attempt as a lost connection. When the instance was still RUNNING, it refreshed the SSH config and ran the same remote command again. A command that exited non-zero and a session that dropped after the command had started both took that path. The second run is a new process, so a long job or an @script starts over while the first one may still be writing the same output.

The first attempt is the execution. It is retried only when ssh's own log shows the session never started. That log is taken with ssh -E, so it is not mixed with the remote command's stderr, and it is still printed after ssh returns. A last non-empty line that starts with ssh: connect to host , ssh: Could not resolve hostname, or kex_exchange_identification: still waits for SSH and runs the command once. That covers a translated strerror under the connect prefix, and a multi-address failure whose last line is still a connect error. A banner failure is the same case written as two lines: kex_exchange_identification: and then Connection closed by <ip|UNKNOWN> port <n> or Connection reset by <ip|UNKNOWN> port <n> from sshpkt_fatal. That second line is retried only when the previous non-empty line is the kex line. Connection closed by 10.0.0.8 port 22 by itself is a drop after the session exists and is not retried. Any other status is returned without a second run. Exit codes other than 255 are the remote command's own status. Exit 255 is not retried when the last line is a drop after the session exists, including Read from remote host …: Connection timed out and Broken pipe, even if an earlier line says Connection refused. An empty diagnostic is not retried either.

Test

go test -count=1 -timeout 180s -run 'TestExecDoesNotReplay|TestExecStillRecovers|TestSSHFailureAllowsReplay|TestRunSSHWithTimeout' ./pkg/cmd/exec

Without the replay check, a remote exit 1 and an exit 255 with Broken pipe both entered SSH recovery and printed Connection failed. With the check, those cases return the original error and invoke the command once. The same holds for Read from remote host brev: Connection timed out and for a log whose first line is ssh: connect to host …: Connection refused and whose last line is Broken pipe. A refused connection whose last line is still ssh: connect to host enters recovery, including when every address failed before a session. The classifier accepts a non-English strerror when it sits under that prefix. A banner close whose lines are kex_exchange_identification: Connection closed by remote host and Connection closed by UNKNOWN port 65535 enters recovery, as does the matching reset pair. A lone Connection closed by 10.0.0.8 port 22 does not, including when a Broken pipe line sits between it and an earlier kex line.

This change was prepared with AI assistance.

brev exec treated every SSH error as a lost connection and, when the
instance was still RUNNING, ran the same command again. A non-zero
remote exit and a session that dropped mid-command both took that path,
so a long job or an @script could start a second time.

Retry only when ssh reports that the session never started. Any other
failure is returned without a second run.
Matching bare errno strings anywhere in the ssh -E log treated a session
that had already started as a failed connect. "Read from remote host:
Connection timed out" and an earlier "Connection refused" followed by
"Broken pipe" both ran the remote command again.

Retry only when the last non-empty diagnostic line starts with an
OpenSSH pre-session prefix.
sshpkt_fatal appends "Connection closed by <ip|UNKNOWN> port <n>" after
kex_exchange_identification logs the banner close. The last line alone
is also a dropped session, so replay only when the previous non-empty
line is that kex prefix.
@pcbeingused333
pcbeingused333 requested a review from a team as a code owner November 14, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

brev exec replays remote command after SSH reconnect

1 participant