Repository navigation
perf(serial): stop paying 220 ms of dead wait on every device RPC - #4343
Conversation
Each request/reply costs two reads in the producer loop: the first
returns the ACK promptly, the second waits for the result and blocks the
full inner cap when the device has not answered yet. The cap was
therefore paid once per RPC, added to the round trip rather than
overlapping it.
Measured on an attached RP2350W over the fbuild monitor, 20 `ping` calls
per point:
cap 0.005 -> 110.5 ms cap 0.10 -> 119.9 ms
cap 0.02 -> 110.1 ms cap 0.25 -> 330.7 ms
cap 0.05 -> 110.2 ms
~110 ms is the genuine round trip. The curve is flat below 0.05 and
steps sharply at 0.25, which is what identifies the cap as dead wait
rather than transport cost -- a `ping` is roughly 4 ms of wire time at
115200, so 330 ms was never the wire.
0.05 keeps the property the cap exists for and improves it: stop_event
is re-checked 20 times a second instead of 4, so cancellation is
strictly more responsive than before.
This is 3x on every serial RPC in AutoResearch, which matters most where
calls are counted in thousands -- the peer-OTA artifact transfer is 3,241
round trips, about 17.8 minutes at 330 ms and about 5.9 at 110.
Verified on hardware: `--rpc-smoke` and `--flex-io --rp-pio-index 0` both
pass unchanged; 329 host tests pass; lint clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 8 included reviews per hour; 3 remain after this review. 📝 WalkthroughWalkthroughThe serial producer thread now caps each inner ChangesSerial read timing
Priority: ➖ Normal Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to The timeout adjustment improves RPC latency and cancellation responsiveness without an identified functional regression. The PR is ready to merge after normal checks. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Follow-up, since this PR's description claims the OTA transfer should drop from ~17.8 min to ~5.9 and I went and tested that claim. The latency win is real and stands — 330.7 ms → 110.2 ms median on But it did not make the peer-OTA run survive, and the reason is that my stated motivation was based on a mechanism I had not checked. I assumed the bench's memory monitor was duration-gated, so a 3x shorter transfer would slip under it. It is not. Across seven attempts the kills land at quite different points:
Two died ~5 minutes in during the build, before any transfer existed to shorten. The monitor fires on So: the 3x is worth having on its own merits (it is 3x on every serial RPC in AutoResearch, and it makes cancellation more responsive), but the "this may unblock #3956's verification" framing in the description was wishful. I am leaving the description's arithmetic in place because it is correct as arithmetic, and adding this so nobody reads it as a demonstrated outcome. #4318's |
|
Direct measurement of the size-independence claim, since the description argued it from a previously-noted figure rather than from data I took myself. Baseline (cap 0.25, i.e. before this change), 10 calls per method:
A residual floor remains that this PR does not address. With the cap at 0.05 the same
Flagging it rather than leaving the impression 110 ms is a floor anyone has justified. If it turns out to be the executor, there is plausibly another sizeable win here for whoever picks it up. Nothing above changes the claim in the description: the 0.25 → 0.05 step is measured, reproducible, and independently verified by |
Correcting the mechanism in this PR's description — the cost is in
|
Tested the structural fix I suggested last comment — it does not workI proposed giving writes their own executor worker so they stop queueing behind the read producer, and said I had not done it because the single-worker design looked deliberate. I have now tested it instead of leaving that as speculation.
So the single worker is load-bearing, not cautious, and the remaining ~100 ms cannot be recovered on the FastLED side by parallelising here. Reverted; latency measures 330 ms again on master as expected. What I can support, and what I cannot. In fbuild's
That shape predicts a concurrent write during a read should be rejected. But the failure I actually observed was a dead liveness probe, not the Where the remaining 100 ms likely lives. Upstream, not here. None of this affects the change in this PR, which stays 330.7 → 110.2 ms, hardware-verified, and now with the dead end documented so nobody re-walks it. |
Widened the regression evidence from two modes to sixI approved-and-moved-on having tested only
Two of these are worth calling out specifically:
What I could not cover
So the wall-clock claim in the description remains arithmetic (3,241 round trips × the measured per-call delta), not a measured end-to-end result, and I would rather say that than leave the six passes above implying more coverage than they give. |
The wall-clock claim is now measured, not projectedI flagged earlier that the "17.8 min → 5.9 min" figure in the description was arithmetic — a single-call delta multiplied by 3,241 — and that I could not run the OTA transfer here to confirm it translates. A sustained bulk workload gets most of the way there, and since this change is host-side only both sides can be measured against the same firmware with no reflash between them. 200 sequential
2.95x, and the per-call figures land on top of the 20-call microbenchmark (330.7 / 110.2 ms). That is the part worth having: the speedup does not decay over a sustained run, so it is not an artifact of isolated calls with a quiet line between them, and the projection to 3,241 round trips is now scaling a measured sustained rate rather than extrapolating from one call. The description's arithmetic was right. It is now backed by a 200-call measurement instead of asserted from a single delta. Still not measured end to end: the actual peer-OTA transfer, which needs both boards built and gets killed by this bench's memory monitor during the build phase. That remains the honest gap, and it is unrelated to this change. |
|
| branch | runs | result |
|---|---|---|
perf/serial-read-cap |
2 | 1 pass (10/10, 362 s), 1 fail at cycle 5 |
master |
3 | 2 pass (10/10, 379 s / 10/10), 1 fail at cycle 7 |
Master fails with the identical signature, so the stall is a pre-existing intermittent property of this fixture, not something this PR introduces:
POST /echo 4096-byte FNV-1a -> status 408, received_bytes: 0
RP server stats: {'timeoutCount': 1, 'lastTimeoutWasStall': True,
'lastTimeoutElapsedMs': 2006, 'lastTimeoutBodyLength': 0,
'lastTimeoutExpectedBody': 4096, 'lastTimeoutHeadersComplete': True}
The RP's HTTP server takes complete headers and then receives none of the 4096-byte body, stalling ~2006 ms into a 408. That is WiFi traffic between the two boards; this PR touches the host-to-device serial control plane. (I also tried to support this with earlier netpeer_fail_*.log files — they turned out to be argument-parse errors, unrelated, so I am not citing them.)
Correction: no speedup on this workload, and I overstated its reach
I said earlier this gate ran "roughly 3x faster per cycle". That is wrong. Measured against the control:
| per cycle | |
|---|---|
| master | 37.9 s |
| this branch | 39.8 s |
Indistinguishable, if anything marginally slower. I had compared against an older --net-peer run with different settings instead of against the control I was about to take.
The reason is that this gate is WiFi-bound, not serial-bound — ~6 s per join plus HTTP transfers dominate, and serial RPCs are a small share. The 2.95x measured over 200 back-to-back calls is real, and it applies to workloads that are actually serial-bound: the OTA artifact transfer is 3,241 sequential round trips and is exactly that shape. This gate is not.
So the description's OTA projection stands, but it should be read as specific to serial-bound work rather than a general speedup of AutoResearch runs.
Net
Regression coverage is now seven gates including the dual-board path, with a control establishing the one failure as pre-existing. The stall itself looks worth tracking separately — it is the server-side analogue of the client-side stall budget raised in #4231, and I do not think it is covered by that fix.
Every AutoResearch serial RPC was costing ~330 ms. About 220 ms of that was the harness waiting on a timeout it did not need to wait on.
Where it goes
FbuildSerialAdapter's producer loop caps each innerread_linessostop_eventgets re-checked. A request/reply costs two reads: the first returns the ACK promptly, the second waits for the result and blocks the full cap when the device has not answered yet. So the cap is paid once per RPC, added to the round trip rather than overlapping it.Measurement
Attached RP2350W (
2DCB876B587EA334) over the fbuild monitor, 20pingcalls per point, median:Two things make this a harness artifact rather than transport cost:
pingis ~4 ms of wire time at 115200. 330 ms was never the wire.Variance is near zero either way (
110 110 110 110 110 110 110 110), which is what you would expect from a constant, and is why this went unnoticed: it looks like a stable, healthy transport.The change
0.25→0.05. That keeps the property the cap exists for and improves it —stop_eventis re-checked 20 times a second instead of 4, so cancellation is strictly more responsive than before, not less. Nothing is gained below 0.05, so this is not shaved to the bone.Why it matters beyond the microbenchmark
3x on every serial RPC in AutoResearch. It matters most where calls are counted in thousands: the peer-OTA artifact transfer is 3,241 round trips — about 17.8 minutes at 330 ms, about 5.9 at 110. That transfer has been the one part of #3956 I could not verify end to end, because every attempt outran this bench's memory monitor.
Verification
bash autoresearch rp2350w --rpc-smoke→RESULT: RPC smoke PASS (...)bash autoresearch rp2350w --flex-io --rp-pio-index 0→TEST PIO0 lanes=1 leds=100 PASS,RESULT PASS 2 test(s)bash lintcleanFound while investigating why the peer-OTA transfer takes ~20 minutes (#3956, #4318).
🤖 Generated with Claude Code
https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51
Summary by CodeRabbit