Repository navigation
Description
Follow-up to #181 / #201. The production test runner now owns compilation, render readiness, screenshots, logs, hard timeouts, authenticated viewer control, and deterministic teardown. This issue adds a host-side command hook after the first real rendered frame.
Goal
Add a repeatable --test-cmd=<COMMAND> option that:
- starts only after the viewer reports
TestEvent::Ready; - runs commands in declaration order, one at a time;
- streams stdout and stderr to the terminal and
--test-log; - shares the existing test's one hard total deadline and Ctrl+C behavior;
- kills the active command and all descendants when the run ends; and
- maps spawn failure or a nonzero command exit to FastLED test exit code
1.
This is a host subprocess hook, not the JavaScript-file injection stretch goal originally sketched in #181. It does not need a new browser API or expose the viewer's bearer capability.
Investigation summary
The required primitives already exist on main:
| Existing capability | Current location | How this issue should reuse it |
|---|---|---|
| Test CLI group and validation | crates/fastled-cli/src/cli.rs, test_mode.rs::build_test_plan |
Add a repeatable Vec<String> and reject empty commands. |
| First-render readiness event | server.rs::TestEvent::Ready, handled by commands.rs::compile_and_test |
Start the command sequence exactly once on the first Ready. |
| One total deadline and Ctrl+C handling | commands.rs::compile_and_test |
Keep these in the outer tokio::select!; do not create a second timeout policy inside the command runner. |
| Timestamped append-only log | commands.rs::write_test_log |
Feed command start/output/exit markers through the same writer. |
| Process-tree containment | running-process 4.0 via ContainedProcessGroup |
Spawn each shell with SpawnStdio::Pipe; dropping SpawnedChild kills its Windows Job Object or Unix process group. |
| Piped child output | running_process::{SpawnStdio, StdioSource::Pipe} |
Drain stdout/stderr concurrently so a noisy child cannot deadlock on a full pipe. |
run_contained_command() is close, but should not be called directly from the Ready match arm: it inherits output and awaits the process, which would stop the outer loop from servicing viewer events and liveness. The post-ready runner needs piped output and must run as a cancel-on-drop future selected alongside viewer events.
Proposed CLI contract
fastled examples/Blink --test \
--test-log=out/run.log \
--test-cmd="python scripts/check_ready.py" \
--test-cmd="python scripts/check_runtime_state.py"Semantics:
- Every occurrence appends one command; clap preserves declaration order.
--test-cmdrequires--test.--test-cmd=and whitespace-only values are rejected before compilation.- Commands run with the sketch directory as their current working directory and inherit the FastLED process environment.
- Stdin is null/non-interactive. No new terminal window is created on Windows.
- Commands use the platform command shell so pipes, redirection, and
&&work:- Unix:
/bin/sh -c <COMMAND> - Windows:
%COMSPEC% /D /S /C <COMMAND>(normallycmd.exe)
- Unix:
- Shell syntax and quoting are therefore platform-specific. On Windows this is
cmd.exesyntax even when the outer invocation came from PowerShell; users can explicitly invokepowershell -NoProfile -Command ...if needed. - Commands are trusted input. They have the user's filesystem/environment permissions and may appear in OS process inspection.
- The command sequence begins at ready and runs concurrently with the viewer's existing wait/capture schedule. The CLI does not report success until both the viewer schedule and every command have completed.
Required lifecycle/state-machine change
Today compile_and_test() returns as soon as it receives browser Done(0). With post-ready commands, browser completion and command completion are independent:
compile -> launch viewer -> Ready
|-- browser wait/captures -> ViewerDone
`-- cmd 0 -> cmd 1 -> ... -> CommandsDone
success evaluation happens only after ViewerDone + CommandsDone
Recommended host state:
ready: boolviewer_done: Option<u8>commands_started: boolcommands_done: bool(initially true only when no commands were supplied)- one cancel-on-drop command task/future plus a bounded command-output channel
On the first Ready, set ready, write the existing ready marker, and start the sequence once. A duplicate ready event must not restart commands.
On browser Done, validate readiness, viewer code, and screenshot count as today, then record viewer_done instead of immediately returning if commands remain. If commands finish first, record commands_done and continue servicing viewer events. When both sides are done, apply the existing page-error rule (2 only for an otherwise successful run with --test-exit-on-error).
Failure precedence:
- Existing hard total timeout remains
124; Ctrl+C remains130; ready timeout remains125. - Viewer/spawn/I/O failure or any nonzero command exit is
1. - Page errors are
2only when viewer and commands otherwise succeeded. - Otherwise return
0.
The command task must not be detached. Use a JoinSet/abort-on-drop guard (or select a pinned owned future directly) so every return path drops the active SpawnedChild. This is what preserves descendant cleanup on timeout, Ctrl+C, viewer failure, or log-write failure.
Command runner design
Add a focused runner in test_mode.rs:
- For each command string, construct the platform shell command and set
current_dir(sketch_dir). - Create
ContainedProcessGroup::with_originator("FASTLED_TEST_CMD"). - Spawn with null stdin and piped stdout/stderr. Keep the default hidden-console and bounded-drain behavior.
- Drain both pipes concurrently. A bounded
tokio::sync::mpscchannel (for example, 256 entries) plus blocking readers gives backpressure instead of unbounded memory growth. - Preserve ordering within each stream; global interleaving between stdout and stderr is naturally nondeterministic.
- After the child exits, drain both readers before emitting the command's completion result so the last log lines cannot appear after the exit marker.
- Stop immediately on spawn failure or nonzero exit; do not start later commands.
Suggested log records (the existing writer adds the millisecond timestamp):
[fastled-test-cmd 0] start
[fastled-test-cmd 0 stdout] server is reachable
[fastled-test-cmd 0 stderr] diagnostic text
[fastled-test-cmd 0] exit=0
Do not include the full command string in the log marker by default because it may contain credentials. Command stderr must not be passed through is_viewer_error_line(); its exit status, not the word error, determines failure. When --test-log is absent, stdout/stderr must still be drained and mirrored to the corresponding terminal streams.
Files to change
| File | Required change |
|---|---|
crates/fastled-cli/src/cli.rs |
Add repeatable test_cmd: Vec<String> under “Production testing,” requires = "test", plus parse/order/help tests. |
crates/fastled-cli/src/test_mode.rs |
Add commands to TestPlan; validate nonempty values; add platform-shell construction, contained sequential execution, piped-output events, and focused tests. |
crates/fastled-cli/src/commands.rs |
Start the command runner on first Ready; select command output/completion alongside viewer events; wait for both viewer and commands; route output through terminal + write_test_log; preserve existing exit precedence and cancellation. |
crates/fastled-cli/src/lib.rs |
Update the manually constructed Cli test fixture with an empty command vector. No routing change is otherwise needed. |
README.md |
Add repeatable examples, working-directory behavior, Unix/Windows shell and quoting rules, output/log format, trusted-command warning, and exit behavior. |
No changes should be needed in server.rs, tauri_viewer.rs, viewer.rs, the authenticated HTTP endpoints, or the browser test runtime. No new crate dependency is expected: tokio and running-process already provide the needed task/channel/process primitives.
Test plan
Focused Rust tests:
- CLI preserves repeated
--test-cmddeclaration order and requires--test. - Empty/whitespace commands fail validation.
- Platform command construction uses
/bin/sh -cor%COMSPEC% /D /S /C, null stdin, and the sketch working directory. - Two commands append
AthenBto a temp file, proving sequential ordering. - stdout and stderr are both captured, tagged with command index/stream, and fully drained before completion.
- A nonzero second command reports failure and prevents a third command from starting.
- Browser
Donebefore command completion waits; command completion before browserDonealso waits. - A total-timeout cancellation drops a long-running command and its descendant process; no delayed sentinel file is created.
- Command stderr containing
errorwith exit0does not become page-error exit2.
End-to-end matrix on the shipped Tauri path:
- A post-ready command can observe that the sketch has rendered, emits stdout/stderr into
--test-log, and exits0. - Two commands execute in order.
- A nonzero command returns FastLED exit
1and later commands do not run. - A hanging command is killed by
--test-timeout-secsand returns124with no surviving child/grandchild. - Ctrl+C returns
130and leaves no child/grandchild. - Existing Blink screenshot, interval naming, page-error
2, ready-timeout125, and NoisePlusPalette WebGL tests remain green. bash lintandsoldr cargo test --workspacepass.
Acceptance criteria
-
--test-cmd=<COMMAND>is repeatable, requires--test, and commands execute once in declaration order after the first rendered-frame ready event. - Commands run from the sketch directory through the documented platform shell with null stdin.
- stdout and stderr are always drained and mirrored to the terminal; when
--test-logis set, timestamped, stream-tagged records and start/exit markers are appended. - Browser completion waits for the command sequence, and command completion waits for browser completion.
- Spawn failure or any nonzero command exit returns
1and prevents later commands from starting. - The existing total timeout (
124), ready timeout (125), page error (2), and Ctrl+C (130) meanings remain stable. - Timeout, Ctrl+C, viewer failure, and early return kill the active shell and every descendant; post-run process audit finds zero survivors.
- Command stderr alone does not count as a viewer/page error.
- No new HTTP endpoint or JavaScript/Tauri injection surface is introduced.
- README documents examples, shell/quoting differences, working directory, trust boundary, logs, and exit behavior.
- Focused tests,
bash lint, andsoldr cargo test --workspacepass, and the existing real Blink/NoisePlusPalette acceptance matrix remains green.
Activity
feat(cli): add post-ready test command hooks
feat(cli): add post-ready test command hooks (#202)