Skip to content

perf(serial): stop paying 220 ms of dead wait on every device RPC - #4343

Merged
zackees merged 1 commit into
masterfrom
perf/serial-read-cap
Sep 12, 2026
Merged

zackees merged 1 commit into
masterfrom
perf/serial-read-cap

Conversation

@zackees

@zackees zackees commented Sep 11, 2026 •

Copy link
Copy Markdown
Member

Every AutoResearch serial RPC was costing ~330 ms. About 220 ms of that was the harness waiting on a timeout it did not need to wait on.

Where it goes

FbuildSerialAdapter's producer loop caps each inner read_lines so stop_event gets re-checked. A request/reply costs two reads: the first returns the ACK promptly, the second waits for the result and blocks the full cap when the device has not answered yet. So the cap is paid once per RPC, added to the round trip rather than overlapping it.

Measurement

Attached RP2350W (2DCB876B587EA334) over the fbuild monitor, 20 ping calls per point, median:

inner cap latency
0.005 s 110.5 ms
0.02 s 110.1 ms
0.05 s 110.2 ms
0.10 s 119.9 ms
0.25 s (before) 330.7 ms

Two things make this a harness artifact rather than transport cost:

  • A ping is ~4 ms of wire time at 115200. 330 ms was never the wire.
  • The curve is a step, not a slope. Flat at ~110 ms for every cap ≤ 0.05, then a sharp jump at 0.25. A real transport cost would scale with payload; this does not — the original figure was already known to be "~318 ms, size-independent", which is the signature of a fixed wait.

Variance is near zero either way (110 110 110 110 110 110 110 110), which is what you would expect from a constant, and is why this went unnoticed: it looks like a stable, healthy transport.

The change

0.25 → 0.05. That keeps the property the cap exists for and improves it — stop_event is re-checked 20 times a second instead of 4, so cancellation is strictly more responsive than before, not less. Nothing is gained below 0.05, so this is not shaved to the bone.

Why it matters beyond the microbenchmark

3x on every serial RPC in AutoResearch. It matters most where calls are counted in thousands: the peer-OTA artifact transfer is 3,241 round trips — about 17.8 minutes at 330 ms, about 5.9 at 110. That transfer has been the one part of #3956 I could not verify end to end, because every attempt outran this bench's memory monitor.

Verification

  • bash autoresearch rp2350w --rpc-smoke → RESULT: RPC smoke PASS (...)
  • bash autoresearch rp2350w --flex-io --rp-pio-index 0 → TEST PIO0 lanes=1 leds=100 PASS, RESULT PASS 2 test(s)
  • 329 host tests pass, 3 skipped
  • bash lint clean

Found while investigating why the peer-OTA transfer takes ~20 minutes (#3956, #4318).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51

Summary by CodeRabbit

  • Bug Fixes
    • Improved serial communication responsiveness by reducing the maximum wait time when reading incoming lines.

Each request/reply costs two reads in the producer loop: the first
returns the ACK promptly, the second waits for the result and blocks the
full inner cap when the device has not answered yet. The cap was
therefore paid once per RPC, added to the round trip rather than
overlapping it.

Measured on an attached RP2350W over the fbuild monitor, 20 `ping` calls
per point:

    cap 0.005 -> 110.5 ms      cap 0.10 -> 119.9 ms
    cap 0.02  -> 110.1 ms      cap 0.25 -> 330.7 ms
    cap 0.05  -> 110.2 ms

~110 ms is the genuine round trip. The curve is flat below 0.05 and
steps sharply at 0.25, which is what identifies the cap as dead wait
rather than transport cost -- a `ping` is roughly 4 ms of wire time at
115200, so 330 ms was never the wire.

0.05 keeps the property the cap exists for and improves it: stop_event
is re-checked 20 times a second instead of 4, so cancellation is
strictly more responsive than before.

This is 3x on every serial RPC in AutoResearch, which matters most where
calls are counted in thousands -- the peer-OTA artifact transfer is 3,241
round trips, about 17.8 minutes at 330 ms and about 5.9 at 110.

Verified on hardware: `--rpc-smoke` and `--flex-io --rp-pio-index 0` both
pass unchanged; 329 host tests pass; lint clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51
@coderabbitai

coderabbitai Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 9da3b4ee-1564-465c-954a-07946d1a3585

📥 Commits

Reviewing files that changed from the base of the PR and between 999304d and ffc389a.

📒 Files selected for processing (1)
  • ci/util/serial_interface.py

Included review availability: Your plan provides up to 8 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The serial producer thread now caps each inner read_lines call at 0.05 seconds instead of 0.25 seconds. The related comment documents the latency and cancellation trade-offs.

Changes

Serial read timing

Layer / File(s) Summary
Update serial read timeout cap
ci/util/serial_interface.py
The producer reduces the inner read_lines timeout cap from 0.25 seconds to 0.05 seconds. The comment explains the timing rationale.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to ffc38

The timeout adjustment improves RPC latency and cancellation responsiveness without an identified functional regression. The PR is ready to merge after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: reducing unnecessary wait time in serial device RPCs. It is concise, specific, and relevant to the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/serial-read-cap

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Follow-up, since this PR's description claims the OTA transfer should drop from ~17.8 min to ~5.9 and I went and tested that claim.

The latency win is real and stands — 330.7 ms → 110.2 ms median on ping, measured the same way before and after, and both --rpc-smoke and PIO0 pass unchanged.

But it did not make the peer-OTA run survive, and the reason is that my stated motivation was based on a mechanism I had not checked. I assumed the bench's memory monitor was duration-gated, so a 3x shorter transfer would slip under it. It is not. Across seven attempts the kills land at quite different points:

attempt log lines at kill stage
ota_attr2 380 build/deploy
ota_318 380 build/deploy
ota_ledger 447 transfer
ota_fast (this fix) 447 transfer

Two died ~5 minutes in during the build, before any transfer existed to shorten. The monitor fires on MemFree dipping — build I/O filling the page cache on a swapless machine — not on elapsed time, so shortening the phase after the build does not address it.

So: the 3x is worth having on its own merits (it is 3x on every serial RPC in AutoResearch, and it makes cancellation more responsive), but the "this may unblock #3956's verification" framing in the description was wishful. I am leaving the description's arithmetic in place because it is correct as arithmetic, and adding this so nobody reads it as a demonstrated outcome.

#4318's 2m staging-region claim therefore stays exactly as labelled there: argued from the device's own ERROR[4]: Not Enough Space, from Updater.cpp:101/:121, and from 524,288 B against a 1,030,348 B image — not demonstrated by a green run.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Direct measurement of the size-independence claim, since the description argued it from a previously-noted figure rather than from data I took myself. Baseline (cap 0.25, i.e. before this change), 10 calls per method:

method median response chars
ping 330.7 ms 140
status 330.6 ms 128
rpc.discover 424.8 ms 4783

ping and status come out identical to 0.1 ms on different payloads, which is the fixed cost this PR removes. And a 34x larger response costs only +94 ms — far less than 115200 baud would permit for 4,643 extra characters (~400 ms), confirming what the wire-time argument implied: USB CDC ignores the configured baud, so the link was never the constraint.

A residual floor remains that this PR does not address. With the cap at 0.05 the same ping sits at ~110 ms, and I have not identified what that is. Two candidates I did not get far enough to separate:

  • the adapter's _executor is single-worker, so the write and the read producer serialise through it, and a producer torn down by stop_event still has to unwind before the next call's producer starts
  • a device-side USB-CDC flush interval, which would be outside the harness entirely

Flagging it rather than leaving the impression 110 ms is a floor anyone has justified. If it turns out to be the executor, there is plausibly another sizeable win here for whoever picks it up.

Nothing above changes the claim in the description: the 0.25 → 0.05 step is measured, reproducible, and independently verified by --rpc-smoke and PIO0 passing on hardware.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Correcting the mechanism in this PR's description — the cost is in write, not the read

I instrumented the adapter rather than leaving my earlier explanation standing, and it was wrong. The numbers and the fix are unaffected; the account of why was not.

What I claimed: a request/reply costs two reads, and the second blocks the full cap waiting for a result that has not arrived.

What actually happens: the cap is paid by the next call's write, which queues on the single-worker _executor behind the previous call's _producer. That producer sits inside read_lines(timeout=inner) and only re-checks stop_event when that inner read returns — so the write waits out the tail of it.

Per-call trace, cap 0.25 (before):

[trace] write took   8.5ms     <- first call, no prior producer to queue behind
[trace] write took 321.0ms
[trace] write took 321.0ms
[trace] write took 320.9ms
n=4  median=330.6ms

and cap 0.05 (after):

[trace] write took  43.8ms     <- first call again
[trace] write took 100.3ms
[trace] write took 100.7ms
[trace] write took 100.4ms
n=4  median=110.3ms

The write is ~91% of the total latency, and the first call in each session is fast precisely because there is no producer in flight yet — which is the signature of queueing, not of waiting on the device. submit -> first-item inside read_lines is a steady 51.4 ms and does not move with the cap, so the read side was never where the time went.

Both points fit write ≈ 1.1 × cap + ~45 ms. The model breaks below 0.05 — cap 0.005 measures the same ~110 ms as cap 0.05 — so a second, unrelated ~100 ms floor takes over there. That is consistent with the flat region in the table above and is why 0.05 is the right stopping point rather than something smaller.

What this implies for the residual floor

My previous comment listed "single-worker executor" and "device-side CDC flush" as unseparated candidates for the ~110 ms. The trace settles it in favour of the executor: the device is not in the picture during those 100 ms, the write has not even been issued yet.

So the structural fix is plausibly to stop writes queueing behind the read producer at all — a separate executor worker, or a producer that can be interrupted rather than polled out. I have not done it because it turns on whether fbuild's SerialMonitor is safe to write to while a read is in flight, and the single-worker design and its stop_event comment look deliberate about exactly that. Someone who knows that invariant could likely take most of the remaining 100 ms.

The change in this PR stands unchanged: 330.7 → 110.2 ms, verified on hardware, and it is now attributed to the right mechanism.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Tested the structural fix I suggested last comment — it does not work

I proposed giving writes their own executor worker so they stop queueing behind the read producer, and said I had not done it because the single-worker design looked deliberate. I have now tested it instead of leaving that as speculation.

ThreadPoolExecutor(max_workers=1) → max_workers=2 breaks the transport outright. The session no longer comes up:

RpcError: Connected to /dev/ttyACM0 but it never answered a liveness probe
(ping, 5s), or the request could not be written.

So the single worker is load-bearing, not cautious, and the remaining ~100 ms cannot be recovered on the FastLED side by parallelising here. Reverted; latency measures 330 ms again on master as expected.

What I can support, and what I cannot. In fbuild's crates/fbuild-python/src/serial_monitor.rs:

  • fn write(&self, data: &str) -> usize — immutable borrow
  • fn read_lines(&mut self, py, timeout) — mutable borrow, held for the whole call including the blocking wait (py.detach releases the GIL, not the borrow)
  • fn read_lines_inner(&self, timeout) already exists, commented "for write_json_rpc which has &self"

That shape predicts a concurrent write during a read should be rejected. But the failure I actually observed was a dead liveness probe, not the Already borrowed that prediction implies — so I am reporting the empirical result and the source signatures, and explicitly not claiming I have established the mechanism. It may be pyo3 borrow semantics; it may be something about monitor state across threads; I did not separate them.

Where the remaining 100 ms likely lives. Upstream, not here. read_lines_inner(&self, ...) demonstrates an &self read path is already viable — the monitor's shared state is behind Mutex (ws_read.lock()) rather than relying on the &mut. If the public read_lines could take &self too, a write would not have to wait out the producer's current inner read at all, and the cap would stop mattering. That is an fbuild change and I am not making it blind from here.

None of this affects the change in this PR, which stays 330.7 → 110.2 ms, hardware-verified, and now with the dead end documented so nobody re-walks it.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Widened the regression evidence from two modes to six

I approved-and-moved-on having tested only --rpc-smoke and PIO0, which is thin for a change that touches every serial exchange in the harness. Re-ran the single-board gates on this branch:

gate result
--rpc-smoke PASS — discovery, help, ping, payload, status, drivers, testNoSerial, RP concurrency, PIO contention, PIO0+PIO1 resources, error handling
--flex-io --rp-pio-index 0 TEST PIO0 lanes=1 leds=100 PASS, RESULT PASS 2 test(s)
--simd SIMD TEST PASSED (85 tests)
--ieee754 IEEE754 CODEC TEST PASSED (35 checks)
--watchdog-soak RESULT: watchdog soak PASS (ack, disconnect, CDC recovery, ping, RPC smoke)
--perf-wave2d 64x64 669.2 / 653.5 fps

Two of these are worth calling out specifically:

--watchdog-soak is the one I most wanted. It deliberately drops and reacquires the CDC link, which is exactly where changing the read-poll cadence 5x could have introduced a race. It passes, including the CDC-recovery leg.

--perf-wave2d lands within noise of the pre-change baseline — 669.2 / 653.5 against 668.6 / 653.6 measured earlier on master. That is the control: this is a host-side change, so a device-side benchmark moving would have meant something unintended.

What I could not cover

--net-peer is the heaviest RPC consumer here (10 reconnect cycles, thousands of calls) and would have been the best evidence, including whether the 3x shows up in wall-clock on a real workload rather than only in a ping microbenchmark. It does not run on this bench. Both boards have to be built, which is the memory-heavy phase, and the run was killed at 232 log lines with zero cycles completed — same monitor behaviour documented above.

So the wall-clock claim in the description remains arithmetic (3,241 round trips × the measured per-call delta), not a measured end-to-end result, and I would rather say that than leave the six passes above implying more coverage than they give.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

The wall-clock claim is now measured, not projected

I flagged earlier that the "17.8 min → 5.9 min" figure in the description was arithmetic — a single-call delta multiplied by 3,241 — and that I could not run the OTA transfer here to confirm it translates. A sustained bulk workload gets most of the way there, and since this change is host-side only both sides can be measured against the same firmware with no reflash between them.

200 sequential ping calls in one session, same board, same firmware, only ci/util/serial_interface.py differing:

200 calls per call projected 3,241
master (cap 0.25) 65.78 s 328.9 ms 17.8 min
this branch (cap 0.05) 22.27 s 111.3 ms 6.0 min

2.95x, and the per-call figures land on top of the 20-call microbenchmark (330.7 / 110.2 ms). That is the part worth having: the speedup does not decay over a sustained run, so it is not an artifact of isolated calls with a quiet line between them, and the projection to 3,241 round trips is now scaling a measured sustained rate rather than extrapolating from one call.

The description's arithmetic was right. It is now backed by a 200-call measurement instead of asserted from a single delta.

Still not measured end to end: the actual peer-OTA transfer, which needs both boards built and gets killed by this bench's memory monitor during the build phase. That remains the honest gap, and it is unrelated to this change.

@zackees

zackees commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

--net-peer regression coverage — and a correction to my speedup claim

This was the one gate I could not run when I widened the regression sweep. Memory conditions allowed it, so here it is, with a control.

First result looked bad for this PR: the branch failed at cycle 5 of 10. Master then passed 10/10. One-and-one is not evidence, and "my change is in a different subsystem" is not an argument, so I repeated both.

branch runs result
perf/serial-read-cap 2 1 pass (10/10, 362 s), 1 fail at cycle 5
master 3 2 pass (10/10, 379 s / 10/10), 1 fail at cycle 7

Master fails with the identical signature, so the stall is a pre-existing intermittent property of this fixture, not something this PR introduces:

POST /echo 4096-byte FNV-1a -> status 408, received_bytes: 0
RP server stats: {'timeoutCount': 1, 'lastTimeoutWasStall': True,
                  'lastTimeoutElapsedMs': 2006, 'lastTimeoutBodyLength': 0,
                  'lastTimeoutExpectedBody': 4096, 'lastTimeoutHeadersComplete': True}

The RP's HTTP server takes complete headers and then receives none of the 4096-byte body, stalling ~2006 ms into a 408. That is WiFi traffic between the two boards; this PR touches the host-to-device serial control plane. (I also tried to support this with earlier netpeer_fail_*.log files — they turned out to be argument-parse errors, unrelated, so I am not citing them.)

Correction: no speedup on this workload, and I overstated its reach

I said earlier this gate ran "roughly 3x faster per cycle". That is wrong. Measured against the control:

per cycle
master 37.9 s
this branch 39.8 s

Indistinguishable, if anything marginally slower. I had compared against an older --net-peer run with different settings instead of against the control I was about to take.

The reason is that this gate is WiFi-bound, not serial-bound — ~6 s per join plus HTTP transfers dominate, and serial RPCs are a small share. The 2.95x measured over 200 back-to-back calls is real, and it applies to workloads that are actually serial-bound: the OTA artifact transfer is 3,241 sequential round trips and is exactly that shape. This gate is not.

So the description's OTA projection stands, but it should be read as specific to serial-bound work rather than a general speedup of AutoResearch runs.

Net

Regression coverage is now seven gates including the dual-board path, with a control establishing the one failure as pre-existing. The stall itself looks worth tracking separately — it is the server-side analogue of the client-side stall budget raised in #4231, and I do not think it is covered by that fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant