Skip to content

feat(cli): add --test-cmd post-ready command hook #200

Closed

Description

@zackees

Follow-up to #181 / #201. The production test runner now owns compilation, render readiness, screenshots, logs, hard timeouts, authenticated viewer control, and deterministic teardown. This issue adds a host-side command hook after the first real rendered frame.

Goal

Add a repeatable --test-cmd=<COMMAND> option that:

  • starts only after the viewer reports TestEvent::Ready;
  • runs commands in declaration order, one at a time;
  • streams stdout and stderr to the terminal and --test-log;
  • shares the existing test's one hard total deadline and Ctrl+C behavior;
  • kills the active command and all descendants when the run ends; and
  • maps spawn failure or a nonzero command exit to FastLED test exit code 1.

This is a host subprocess hook, not the JavaScript-file injection stretch goal originally sketched in #181. It does not need a new browser API or expose the viewer's bearer capability.

Investigation summary

The required primitives already exist on main:

Existing capability Current location How this issue should reuse it
Test CLI group and validation crates/fastled-cli/src/cli.rs, test_mode.rs::build_test_plan Add a repeatable Vec<String> and reject empty commands.
First-render readiness event server.rs::TestEvent::Ready, handled by commands.rs::compile_and_test Start the command sequence exactly once on the first Ready.
One total deadline and Ctrl+C handling commands.rs::compile_and_test Keep these in the outer tokio::select!; do not create a second timeout policy inside the command runner.
Timestamped append-only log commands.rs::write_test_log Feed command start/output/exit markers through the same writer.
Process-tree containment running-process 4.0 via ContainedProcessGroup Spawn each shell with SpawnStdio::Pipe; dropping SpawnedChild kills its Windows Job Object or Unix process group.
Piped child output running_process::{SpawnStdio, StdioSource::Pipe} Drain stdout/stderr concurrently so a noisy child cannot deadlock on a full pipe.

run_contained_command() is close, but should not be called directly from the Ready match arm: it inherits output and awaits the process, which would stop the outer loop from servicing viewer events and liveness. The post-ready runner needs piped output and must run as a cancel-on-drop future selected alongside viewer events.

Proposed CLI contract

fastled examples/Blink --test \
  --test-log=out/run.log \
  --test-cmd="python scripts/check_ready.py" \
  --test-cmd="python scripts/check_runtime_state.py"

Semantics:

  • Every occurrence appends one command; clap preserves declaration order.
  • --test-cmd requires --test. --test-cmd= and whitespace-only values are rejected before compilation.
  • Commands run with the sketch directory as their current working directory and inherit the FastLED process environment.
  • Stdin is null/non-interactive. No new terminal window is created on Windows.
  • Commands use the platform command shell so pipes, redirection, and && work:
    • Unix: /bin/sh -c <COMMAND>
    • Windows: %COMSPEC% /D /S /C <COMMAND> (normally cmd.exe)
  • Shell syntax and quoting are therefore platform-specific. On Windows this is cmd.exe syntax even when the outer invocation came from PowerShell; users can explicitly invoke powershell -NoProfile -Command ... if needed.
  • Commands are trusted input. They have the user's filesystem/environment permissions and may appear in OS process inspection.
  • The command sequence begins at ready and runs concurrently with the viewer's existing wait/capture schedule. The CLI does not report success until both the viewer schedule and every command have completed.

Required lifecycle/state-machine change

Today compile_and_test() returns as soon as it receives browser Done(0). With post-ready commands, browser completion and command completion are independent:

compile -> launch viewer -> Ready
                            |-- browser wait/captures -> ViewerDone
                            `-- cmd 0 -> cmd 1 -> ... -> CommandsDone

success evaluation happens only after ViewerDone + CommandsDone

Recommended host state:

  • ready: bool
  • viewer_done: Option<u8>
  • commands_started: bool
  • commands_done: bool (initially true only when no commands were supplied)
  • one cancel-on-drop command task/future plus a bounded command-output channel

On the first Ready, set ready, write the existing ready marker, and start the sequence once. A duplicate ready event must not restart commands.

On browser Done, validate readiness, viewer code, and screenshot count as today, then record viewer_done instead of immediately returning if commands remain. If commands finish first, record commands_done and continue servicing viewer events. When both sides are done, apply the existing page-error rule (2 only for an otherwise successful run with --test-exit-on-error).

Failure precedence:

  1. Existing hard total timeout remains 124; Ctrl+C remains 130; ready timeout remains 125.
  2. Viewer/spawn/I/O failure or any nonzero command exit is 1.
  3. Page errors are 2 only when viewer and commands otherwise succeeded.
  4. Otherwise return 0.

The command task must not be detached. Use a JoinSet/abort-on-drop guard (or select a pinned owned future directly) so every return path drops the active SpawnedChild. This is what preserves descendant cleanup on timeout, Ctrl+C, viewer failure, or log-write failure.

Command runner design

Add a focused runner in test_mode.rs:

  1. For each command string, construct the platform shell command and set current_dir(sketch_dir).
  2. Create ContainedProcessGroup::with_originator("FASTLED_TEST_CMD").
  3. Spawn with null stdin and piped stdout/stderr. Keep the default hidden-console and bounded-drain behavior.
  4. Drain both pipes concurrently. A bounded tokio::sync::mpsc channel (for example, 256 entries) plus blocking readers gives backpressure instead of unbounded memory growth.
  5. Preserve ordering within each stream; global interleaving between stdout and stderr is naturally nondeterministic.
  6. After the child exits, drain both readers before emitting the command's completion result so the last log lines cannot appear after the exit marker.
  7. Stop immediately on spawn failure or nonzero exit; do not start later commands.

Suggested log records (the existing writer adds the millisecond timestamp):

[fastled-test-cmd 0] start
[fastled-test-cmd 0 stdout] server is reachable
[fastled-test-cmd 0 stderr] diagnostic text
[fastled-test-cmd 0] exit=0

Do not include the full command string in the log marker by default because it may contain credentials. Command stderr must not be passed through is_viewer_error_line(); its exit status, not the word error, determines failure. When --test-log is absent, stdout/stderr must still be drained and mirrored to the corresponding terminal streams.

Files to change

File Required change
crates/fastled-cli/src/cli.rs Add repeatable test_cmd: Vec<String> under “Production testing,” requires = "test", plus parse/order/help tests.
crates/fastled-cli/src/test_mode.rs Add commands to TestPlan; validate nonempty values; add platform-shell construction, contained sequential execution, piped-output events, and focused tests.
crates/fastled-cli/src/commands.rs Start the command runner on first Ready; select command output/completion alongside viewer events; wait for both viewer and commands; route output through terminal + write_test_log; preserve existing exit precedence and cancellation.
crates/fastled-cli/src/lib.rs Update the manually constructed Cli test fixture with an empty command vector. No routing change is otherwise needed.
README.md Add repeatable examples, working-directory behavior, Unix/Windows shell and quoting rules, output/log format, trusted-command warning, and exit behavior.

No changes should be needed in server.rs, tauri_viewer.rs, viewer.rs, the authenticated HTTP endpoints, or the browser test runtime. No new crate dependency is expected: tokio and running-process already provide the needed task/channel/process primitives.

Test plan

Focused Rust tests:

  • CLI preserves repeated --test-cmd declaration order and requires --test.
  • Empty/whitespace commands fail validation.
  • Platform command construction uses /bin/sh -c or %COMSPEC% /D /S /C, null stdin, and the sketch working directory.
  • Two commands append A then B to a temp file, proving sequential ordering.
  • stdout and stderr are both captured, tagged with command index/stream, and fully drained before completion.
  • A nonzero second command reports failure and prevents a third command from starting.
  • Browser Done before command completion waits; command completion before browser Done also waits.
  • A total-timeout cancellation drops a long-running command and its descendant process; no delayed sentinel file is created.
  • Command stderr containing error with exit 0 does not become page-error exit 2.

End-to-end matrix on the shipped Tauri path:

  • A post-ready command can observe that the sketch has rendered, emits stdout/stderr into --test-log, and exits 0.
  • Two commands execute in order.
  • A nonzero command returns FastLED exit 1 and later commands do not run.
  • A hanging command is killed by --test-timeout-secs and returns 124 with no surviving child/grandchild.
  • Ctrl+C returns 130 and leaves no child/grandchild.
  • Existing Blink screenshot, interval naming, page-error 2, ready-timeout 125, and NoisePlusPalette WebGL tests remain green.
  • bash lint and soldr cargo test --workspace pass.

Acceptance criteria

  • --test-cmd=<COMMAND> is repeatable, requires --test, and commands execute once in declaration order after the first rendered-frame ready event.
  • Commands run from the sketch directory through the documented platform shell with null stdin.
  • stdout and stderr are always drained and mirrored to the terminal; when --test-log is set, timestamped, stream-tagged records and start/exit markers are appended.
  • Browser completion waits for the command sequence, and command completion waits for browser completion.
  • Spawn failure or any nonzero command exit returns 1 and prevents later commands from starting.
  • The existing total timeout (124), ready timeout (125), page error (2), and Ctrl+C (130) meanings remain stable.
  • Timeout, Ctrl+C, viewer failure, and early return kill the active shell and every descendant; post-run process audit finds zero survivors.
  • Command stderr alone does not count as a viewer/page error.
  • No new HTTP endpoint or JavaScript/Tauri injection surface is introduced.
  • README documents examples, shell/quoting differences, working directory, trust boundary, logs, and exit behavior.
  • Focused tests, bash lint, and soldr cargo test --workspace pass, and the existing real Blink/NoisePlusPalette acceptance matrix remains green.

Activity

  1. added 2 commits that reference this issue on Jul 14, 2026
    f1adf9c
    2465652
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions