Skip to content

feat(cli): built-in production-testing hooks — screenshot, console log capture, auto-exit for agentic verification #181

Closed

Description

@zackees

Context

fastled-wasm compiles FastLED sketches to WASM and shows them in a Tauri-hosted viewer pointed at a local HTTP server (see commit 0332caf and #154 for live build streaming). The user-facing CLI is the Rust binary in crates/fastled-cli. Python src/fastled/ is a thin shim.

The product owner wants lightweight, in-product testing hooks so an AI agent (or CI script) can verify what a compiled sketch actually renders — without bringing in Playwright, a headless Chromium, or any external browser-automation framework. The hooks must reuse the existing viewer process and HTTP server.

The implementing agent should be fable-5 with xhigh reasoning effort.

Motivation — why not Playwright?

  • Playwright requires a separate Node/Chromium toolchain that does not match the Tauri/WebView2/WKWebView runtime the user actually ships. Verifying a different browser is an unsound test.
  • It bloats the install / CI footprint of a project whose value proposition is "one Rust binary, no Python required to run."
  • The viewer process and HTTP server are already in fastled; adding a few flags is far cheaper than an external harness.
  • An AI agent reasoning about a sketch needs a single deterministic command that produces a PNG and a log file, then exits with a clean status code. That maps directly to a flag, not a third-party tool.

Investigation Findings (what already exists)

The plumbing for almost every desired capability is in place:

Capability Status Location
HTTP server with SSE build stream shipped (#154) crates/fastled-cli/src/server.rs
Build-finish status event (status:success) shipped compile_stream.rs::report_build_outcome
Viewer-side console/error/fetch capture shipped behind FASTLED_VIEWER_LOGS=1 tauri_viewer.rs::LOG_FORWARD_SCRIPT
POST /viewer-log endpoint (echoes to stderr) shipped server.rs::viewer_log
WebviewWindow::eval() for JS injection shipped, already used tauri_viewer.rs
Viewer process liveness polling + clean shutdown shipped commands.rs::compile_and_serve
Screenshot API in Tauri 2.10 / wry 0.54 NOT available confirmed in registry sources
Headless mode for the viewer not available; --internal-serve-dir-headless only skips the viewer

Screenshot feasibility: wry 0.54 / tauri 2.10 expose no take_screenshot / capture_preview binding. Going OS-native (WebView2 ICoreWebView2_2::CapturePreview on Windows, WKWebView.takeSnapshot on macOS, GTK on Linux) is feasible but triples the surface area. The portable alternative is HTML5 canvas readback: the FastLED WASM renderer paints into a <canvas>, so canvas.toDataURL("image/png") produces a faithful frame. That PNG is POSTed back through a new /viewer-screenshot endpoint — same pattern as the existing /viewer-log. Recommend canvas readback. OS-native capture can be added later behind the same CLI surface without changing flags.

Proposed CLI API

Recommended: a single --test umbrella flag with structured sub-options

fastled examples/Blink \
    --test \
    --test-wait-secs=2.0 \
    --test-screenshot=out/blink.png \
    --test-log=out/blink.viewer.log \
    --test-exit-on-error

Interval mode:

fastled examples/Blink \
    --test \
    --test-wait-secs=1.0 \
    --test-interval-secs=0.5 \
    --test-count=10 \
    --test-screenshot=out/blink-{n:03}.png \
    --test-log=out/blink.viewer.log
Flag Purpose
--test Master switch. Implies no_interactive, disables file-watch rebuilds, enables canvas-readback + log-file sinks, and arms auto-exit.
--test-wait-secs=<f> Seconds to wait after the "sketch ready" event (compile time excluded). Default 1.0.
--test-screenshot=<PATH> Output PNG path. Supports {n}/{n:03} and {ts} placeholders for interval mode. Required if any screenshot is desired.
--test-interval-secs=<f> Capture cadence. Triggers interval mode.
--test-count=<N> Number of frames in interval mode. Mutually exclusive with --test-duration-secs.
--test-duration-secs=<f> Total capture window in interval mode.
--test-log=<PATH> Append viewer console/error log to file. Implies FASTLED_VIEWER_LOGS=1.
--test-exit-on-error Nonzero exit if any console.error, page error, or failed fetch was observed during the test window.
--test-timeout-secs=<f> Hard upper bound on total test duration (compile + wait + capture). Default 120. Exit code 124 on timeout.
--test-ready-timeout-secs=<f> Time to wait for the "sketch ready" signal after status:success. Default 15. Exit code 125 on ready-timeout.
--test-cmd=<file.js> Stretch goal. Inject a JS file after the ready event.

Alternatives Considered

  1. Flat boolean: --test-wait-then-take-picture-exit (user strawman). Self-documenting but does not extend to interval mode, separate log file, or exit-on-error semantics without a flag explosion.
  2. Subcommand: fastled test <sketch> [...]. Cleanest grouping, best --help discoverability for agents, no collision with existing positional directory. Strong runner-up. Rejected only because the current CLI is flag-driven (no subcommands today) and introducing one is a larger surgical change. If the implementing agent finds clap subcommand wiring straightforward, switching to this is preferred and explicitly endorsed.
  3. JSON config: --test-config=test.json. Keeps the flag surface tiny but is hostile to one-shot agent invocations (an agent now has to write a file first).
  4. Reuse --just-compile + a separate --screenshot-after. Conflates compile-only mode with viewer launch; the viewer must be alive to capture frames. Rejected.

Why the recommended design wins

  • Discoverable: every flag is prefixed --test-* so fastled --help | grep test is exhaustive.
  • Scriptable for agents: a single command produces deterministic artifacts and a meaningful exit code; no orchestration of a long-running process is needed.
  • Backward-compatible: zero collision with existing flags.
  • Extensible: the stretch-goal --test-cmd slots in without re-shaping anything.

Ready-signal Definition (when does the "wait X seconds" clock start?)

The "ready" event is the later of:

  1. SSE build-stream event with status:"success" (compile finished, index.html on disk), AND
  2. Viewer has navigated to the real page (not the loading page) AND a <canvas> exists in the DOM AND at least two requestAnimationFrame ticks have elapsed after canvas appearance.

Detection is JS-side, injected by the host after page load:

window.__fastled_test = window.__fastled_test || {};
const ready = new Promise(resolve => {
  const tryFind = () => {
    const c = document.querySelector('canvas');
    if (c) {
      requestAnimationFrame(() => requestAnimationFrame(() => resolve(c)));
    } else {
      setTimeout(tryFind, 50);
    }
  };
  document.addEventListener('DOMContentLoaded', tryFind);
  if (document.readyState !== 'loading') tryFind();
});
ready.then(c => {
  window.__fastled_test.canvas = c;
  fetch('/test-ready', { method: 'POST' });
});

The host arms its --test-wait-secs timer when it receives POST /test-ready. If that never arrives within --test-ready-timeout-secs, the run exits with code 125. This explicitly excludes compile time from the wait, matching the stated requirement.

Console Log Capture

The existing LOG_FORWARD_SCRIPT is already correct: it intercepts console.{log,info,warn,error,debug}, window.onerror, unhandledrejection, and failed fetch. Adding --test-log=<PATH>:

  • Sets FASTLED_VIEWER_LOGS=1 for the spawned viewer subprocess (already supported).
  • Opens the file in append mode in the host CLI before starting the viewer.
  • Tees viewer_log writes to both stderr (current behavior) and the file. Each line is prefixed with a millisecond timestamp.
  • Records a marker line [fastled-test] ready when /test-ready fires and [fastled-test] screenshot=<path> after each capture, for easy correlation.

--test-exit-on-error scans the tee stream for any line whose console level is error, or that starts with window.onerror: / unhandledrejection: / fetch failed: / fetch error:, and sets the exit code accordingly.

Exit Semantics & Exit Codes

Code Meaning
0 Test ran to completion. No errors observed (or --test-exit-on-error not set).
1 Generic failure (compile failed, viewer failed to launch, IO error writing PNG/log).
2 One or more page-side errors observed with --test-exit-on-error.
124 Hit --test-timeout-secs.
125 Did not observe ready signal within --test-ready-timeout-secs.

Auto-exit semantics:

  • After all screenshots are written, host calls window.close() via eval and then drops ViewerProcess (existing kill-on-job-close on Windows, ContainedProcessGroup on Unix). No lingering window or HTTP server.
  • Ctrl+C remains supported and exits with 130.

Screenshot Capture (implementation)

JS injected after /test-ready:

window.__fastled_test.capture = async function(name) {
  const c = window.__fastled_test.canvas;
  // toDataURL preserves the most recently composited frame for 2D/WebGL with
  // preserveDrawingBuffer:true; for WebGL contexts created without it we
  // request a synchronous readback via gl.readPixels in the same RAF tick.
  const dataUrl = c.toDataURL('image/png');
  await fetch('/viewer-screenshot?name=' + encodeURIComponent(name), {
    method: 'POST',
    headers: {'Content-Type': 'application/octet-stream'},
    body: await (await fetch(dataUrl)).blob()
  });
};

New endpoint POST /viewer-screenshot?name=<basename> in server.rs decodes the body and writes it to the configured output directory. The host's capture scheduler is a simple tokio::time::interval driven by --test-interval-secs; it issues a WebviewWindow::eval("window.__fastled_test.capture('frame-N.png')") per tick.

WebGL caveat (CRITICAL): FastLED's WASM renderer may create its WebGL context without preserveDrawingBuffer: true. In that case toDataURL returns a blank frame. Mitigation: the injected JS detects getContext("webgl"|"webgl2") and patches the constructor to force preserveDrawingBuffer: true BEFORE the sketch's JS runs. Implementing agent must verify this with at least one WebGL-based example (e.g., examples/NoisePlusPalette).

Stretch Goal — --test-cmd <file.js>

When --test-cmd is set, the host reads the JS file, base64-encodes it, and after /test-ready calls eval("(new Function(atob('...')))()"). A minimal host bridge is exposed to the injected script:

window.__fastled_test.screenshot(name);    // forces a screenshot
window.__fastled_test.log(msg);            // routed through /viewer-log
window.__fastled_test.done(exit_code);     // POSTs /test-done, host exits
window.__fastled_test.fail(message);       // shorthand for done(2) + log

POST /test-done is the IPC channel back to the host that completes the run. Without it, an injected script has no way to terminate the host early — the host would always run the full --test-wait-secs then exit.

Why deferred:

  • Requires designing a stable JS-side API surface (versioned via window.__fastled_test.version).
  • Introduces user-controlled JS evaluation against the FastLED page — needs an explicit security note ("only run scripts you trust; injected JS has full DOM access").
  • The non-stretch features above already deliver the verification capability for >=80% of agent workflows (compile -> render -> screenshot -> check log).

Implementation Sketch

Files to touch (all in crates/fastled-cli):

  • src/cli.rs — add the --test* flags. Keep them under a #[command(next_help_heading="Production testing")] group for discoverability.
  • src/lib.rs — early-branch into commands::compile_and_test when --test is set.
  • src/commands.rs — new compile_and_test() derived from compile_and_serve(), with: no file watcher, deterministic shutdown, capture-scheduler task, --test-log tee. Do not duplicate the compile path — refactor the watch loop into a RunMode enum.
  • src/server.rs — add POST /viewer-ready, POST /viewer-screenshot, POST /test-done handlers; pipe their events out via a new mpsc::Sender<TestEvent> field on AppState.
  • src/tauri_viewer.rs — extend ViewerOptions with inject_test_runtime: bool; concatenate a new TEST_RUNTIME_SCRIPT (ready-detect + capture API + WebGL preserveDrawingBuffer patch) onto the existing LOG_FORWARD_SCRIPT.
  • src/main.rs — propagate test-runtime injection through --internal-viewer (new sibling flag --viewer-inject-test-runtime).

Feasibility notes from investigation:

  • WebviewWindow::eval is available in tauri 2.10 and is the only JS injection primitive needed — no IPC plugin required because all paths use HTTP fetch back to the same server. The architecture is symmetric with the existing log-forwarding feature, which is a strong sign it will land cleanly.
  • Tauri/wry have no screenshot binding in the versions in use; canvas readback is the correct choice.
  • Auto-exit already works via ViewerProcess::is_alive polling; we just add a "test-completed" condition alongside the existing "viewer closed" condition.

Acceptance Criteria

  • fastled <sketch> --test --test-wait-secs=2 --test-screenshot=out.png compiles the sketch, waits 2 s after the sketch renders, writes a non-blank PNG, and exits 0. Compile time is not counted toward the 2 s.
  • --test-interval-secs=0.5 --test-count=10 --test-screenshot=out-{n:03}.png writes exactly 10 PNGs named out-000.png ... out-009.png.
  • --test-log=run.log produces a file containing at minimum the [fastled-test] ready marker and every console.* line emitted by the sketch.
  • A sketch that calls console.error("boom") with --test-exit-on-error set exits with code 2.
  • A sketch that fails to compile exits with code 1. A sketch that compiles but never renders a canvas within --test-ready-timeout-secs exits with code 125.
  • No viewer window, viewer process, or HTTP server remains alive after the CLI exits (verified by job-object / process-group teardown).
  • At least one WebGL-based example produces a non-blank screenshot (validates the preserveDrawingBuffer patch).
  • bash lint and soldr cargo test --workspace pass. New unit tests cover: filename templating, exit-code mapping, ready-timeout, error-line classification.
  • Documentation updated in README.md (one short "Testing your sketch from CI / an agent" section) and CLAUDE.md Conventions note pointing at the test flags.
  • --test-cmd is not required for this PR; it lands behind a follow-up issue referencing this one.

Decisions

  • Single --test umbrella with prefixed sub-flags, not a subcommand. Smaller refactor, easier to extend.
  • Canvas toDataURL over OS-native window capture. Portable, no per-platform code, faithful to what the user actually sees in the canvas.
  • JS injection over a Tauri IPC plugin. Reuses the existing LOG_FORWARD_SCRIPT pattern; no new plugin/feature flag on tauri.
  • HTTP POST back to the same server, not Tauri IPC. The server is already there, the viewer already POSTs to it, and this keeps the design uniform.
  • Patch getContext to force preserveDrawingBuffer: true rather than swap WebGL for software rendering. Cheaper, doesn't change what the sketch sees.
  • Defer --test-cmd to a follow-up; the bridge design is sketched here so the follow-up doesn't re-litigate it.
  • Exit-on-error is opt-in. A sketch that logs a benign warning shouldn't fail a screenshot run by default.

Activity

  1. added a commit that references this issue on Jul 13, 2026
    7229055
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions