Skip to content

feat: bundle the Rust fastled CLI binary in the wheel (closes #67) - #68

Merged
zackees merged 4 commits into
mainfrom
feat/bundle-rust-binary
May 13, 2026
Merged

zackees merged 4 commits into
mainfrom
feat/bundle-rust-binary

Conversation

@zackees

@zackees zackees commented May 13, 2026

Copy link
Copy Markdown
Member

Summary

Implements #67. End users who pip install fastled will now receive the Rust fastled binary directly in their venv's Scripts//bin/ — no Python entry-point shim, no cargo build at install time.

Mechanism

maturin's [tool.maturin] data config points at a wheel-data/ directory which maturin packages as the wheel's <pkg>-<ver>.data/ tree. Pip extracts data/scripts/* to the venv's Scripts/bin on install. Verified locally:

$ ./build-wheel
$ unzip -l target/wheels/fastled-2.0.7-cp310-abi3-win_amd64.whl | grep .data
    6303744  fastled-2.0.7.data/scripts/fastled.exe

$ uv pip install target/wheels/fastled-*.whl --python /tmp/test-venv/Scripts/python.exe
$ /tmp/test-venv/Scripts/fastled.exe --version
fastled 2.0.7
$ /tmp/test-venv/Scripts/fastled.exe --help | head -1
FastLED WASM compilation CLI

The Python fastled = "fastled.cli:main" entry-point is gone; the Rust binary takes over that name. fled and fastled-wasm remain as Python aliases for python -m fastled.app-style invocations.

Changes

  • crates/fastled-cli — refactored into [lib] + [[bin]]. lib.rs holds the implementation with pub fn run() -> ExitCode; main.rs is a one-liner that calls fastled_cli::run(). Reusable from other consumers in the future.
  • pyproject.toml — adds [tool.maturin] data = "wheel-data"; drops fastled from [project.scripts].
  • build-wheel (new) — local helper that compiles the bin (via the _cargo trampoline so MSVC target is forced on Windows), stages it in wheel-data/scripts/, then runs maturin build --release.
  • .github/workflows/_build.yml — adds a pre-step that compiles the bin for the target triple and stages it before maturin build. Cross-compile path handled.
  • src/fastled/_rust_cli.py — drops the _looks_like_python_dir PATH filter; with the Python fastled shim gone, plain shutil.which("fastled") is sufficient for wheel-install users.
  • .gitignore — ignore wheel-data/.

Scope

This PR delivers the wheel-bundling foundation. Editable installs (CI unit-test, deploy, local dev) keep using PR #66's cargo build --release --bin fastled fallback in ./install so target/release/fastled is populated for the workspace-target lookup. A future PR can switch CI's editable installs to wheel installs if desired.

Test plan

Closes #67

🤖 Generated with Claude Code

zackees added 4 commits May 12, 2026 20:40
End users who `pip install fastled` will now receive the Rust `fastled`
binary directly in their venv's Scripts/bin directory — no Python
entry-point shim, no cargo build at install time.

Mechanism: maturin's `[tool.maturin] data` config points at a
`wheel-data/` directory which it packages as the wheel's
`<pkg>-<ver>.data/` tree. Pip extracts `data/scripts/*` to the venv's
Scripts/ (Windows) or bin/ (Unix) on install. Verified locally:

    $ ./build-wheel
    $ unzip -l target/wheels/fastled-2.0.7-cp310-abi3-win_amd64.whl | grep .data
        6303744  fastled-2.0.7.data/scripts/fastled.exe

    $ uv pip install target/wheels/fastled-*.whl
    $ which fastled
    /tmp/test-venv/Scripts/fastled.exe        # the Rust binary
    $ fastled --version
    fastled 2.0.7

Changes:

- `crates/fastled-cli/src/lib.rs` (new) — moved the body of main.rs and
  renamed `fn main` to `pub fn run`. `crates/fastled-cli/src/main.rs`
  shrinks to a one-liner calling `fastled_cli::run()`. The crate now
  exposes both `[lib]` and `[[bin]]`, so the binary's logic can also be
  called from other consumers in the future.
- `pyproject.toml` — adds `[tool.maturin] data = "wheel-data"`; drops
  `fastled = "fastled.cli:main"` from `[project.scripts]` (the Rust
  binary takes over that command). `fled` and `fastled-wasm` stay as
  Python aliases.
- `build-wheel` (new) — helper that compiles the bin via the `_cargo`
  trampoline, stages it under `wheel-data/scripts/`, then runs
  `maturin build --release`.
- `.github/workflows/_build.yml` — adds a pre-step that compiles the bin
  for the target triple and stages it under `wheel-data/scripts/` before
  the existing `maturin build` step. Cross-compile path handled.
- `src/fastled/_rust_cli.py` — drops the `_looks_like_python_dir` PATH
  filter; the Python `fastled` shim no longer exists, so plain
  `shutil.which("fastled")` is sufficient for the wheel-install case.
- `.gitignore` — ignore `wheel-data/`.

Editable installs (CI unit-test, deploy, local dev) are unchanged: the
existing cargo-build fallback in `./install` (PR #66) keeps populating
`target/release/fastled` for the workspace-target lookup. A follow-up
can switch CI's editable installs to wheel installs if desired, but
that's orthogonal to this PR.

Local: `bash lint` clean, 119 unit tests pass.
PR #68's `[tool.maturin] data = "wheel-data"` config requires the
directory to exist at maturin invocation time. CI jobs (build,
unit-test, integration-test, deploy) all run `./install` which does
`uv pip install -e .` → maturin pep517 build-wheel; without wheel-data
present, maturin errors with `Caused by: No such data directory`.

Move the directory creation to the top of `./install`, before any
maturin invocation. The cargo-build step later in the same script
populates `wheel-data/scripts/` with the Rust binary for downstream
`./build-wheel` use; an empty dir is sufficient for the editable-install
path that pip drives.
PR #68's deploy job failed with `FileNotFoundError: fastled executable
not found` because `build_site.py` does `which("fastled")` and shells
out to plain `fastled --init=...`. Editable installs (`uv pip install -e
.`) don't extract the maturin `data` dir to the venv's Scripts/, so
PATH lookup found nothing.

After cargo-build (the binstall fallback added in PR #66), copy the Rust
binary into:

* `.venv/Scripts/fastled.exe` (Windows) or `.venv/bin/fastled` (Unix) —
  mimics the wheel-install layout so PATH lookup resolves to the Rust
  binary for editable installs.
* `wheel-data/scripts/fastled[.exe]` — so subsequent `./build-wheel`
  bundles the same binary into the wheel.

Discovery walks the candidate `target/<triple>/release/`,
`target/release/`, and `$CARGO_HOME/bin/` paths so both cargo-build and
cargo-binstall outputs are handled.
@zackees
zackees merged commit 838c8da into main May 13, 2026
24 of 25 checks passed
zackees added a commit that referenced this pull request May 13, 2026
…#69)

* fix(cli): install emscripten + esbuild before --just-compile delegation

PR #68's deploy job (`build_site.py`) now reaches the Rust binary via
PATH, but the next call — `fastled <dir> --just-compile` — falls into
the Python delegation branch of `lib.rs` WITHOUT calling
`install::ensure_emscripten_installed()`. Python's emscripten lookup
returns None on a fresh CI runner and compile fails with:

    Native compilation failed: Emscripten SDK not found.

`compile_and_serve` (the non-just-compile path) does install the
toolchain. Promote the same pre-step to also cover the just-compile
delegation: whenever the CLI is about to invoke `python -m fastled.app`
with a sketch directory (and we're not in `--init` / `--install`
mode), ensure emscripten + esbuild are installed and export
FASTLED_EMSCRIPTEN_DIR / FASTLED_ESBUILD_PATH for the subprocess.

Local: bash lint clean, 119 unit tests pass.

* fix(rust): parse both manifest layouts (nested + inline versions)

The previous deploy failure after PR #69 was a manifest-parse error:

    parse manifest JSON from .../linux/x86_64/manifest.json:
    bad version entry 'versions': missing field `href`

The clang-tool-chain-bins emscripten manifest is served in two shapes:

  * Linux / macOS — nested:
      {"latest": "4.0.21", "versions": {"4.0.21": {...}}}
  * Windows — inline:
      {"latest": "4.0.19", "4.0.19": {...}}

The published `ctcb-manifest` crate's custom deserializer only handles
the inline shape; it fails on Linux. Drop the dep, define local
PartRef/VersionInfo/PlatformManifest types in `install.rs`, and write a
custom Deserialize for PlatformManifest that accepts both shapes
(prefer nested `versions` when present, fall back to inline keys).

Verified locally by running `target/release/fastled <dummy> --just-compile`
against the live Windows manifest — install proceeds past the parse and
the only remaining error is a missing-sketch failure, unrelated to the
toolchain bring-up. CI Linux deploy will now exercise the nested path.

* fix(deps): add meson + ninja to runtime deps

After PR #69's earlier commits got the toolchain pre-step working, the
deploy job advances further into the compile flow before failing:

    [WASM] Build failed: [Errno 2] No such file or directory: 'meson'
    File ".../internal_wasm_build.py", line 600, in ensure_meson_configured

The FastLED C++ → WASM build invokes `meson` via subprocess (and meson
runs `ninja` as its backend), but neither was declared as a dependency
of the fastled package. CI runners don't have them on PATH; pip-install
of fastled didn't provide them either.

Both ship standalone executables when pip-installed (via `meson` and
`ninja` PyPI packages), so adding them to `[project.dependencies]` puts
them in `.venv/Scripts/` and on PATH for the subprocess. Net runtime
deps go from 1 → 3 (`meson`, `ninja`, `zcmds_win32; Windows`), but all
three are load-bearing for the compile critical path.

* fix(deps): re-add clang-tool-chain for the meson wasm cross-file

After meson + ninja landed, the deploy job's compile now reaches meson's
configure step and fails with:

    meson.build:1:0: ERROR: Unknown compiler(s): [['clang-tool-chain-emcc']]
    Running `clang-tool-chain-emcc --version` gave
    "[Errno 2] No such file or directory: 'clang-tool-chain-emcc'"

FastLED's `ci/meson/wasm_cross_file.ini` names `clang-tool-chain-emcc`
/ `clang-tool-chain-em++` / `clang-tool-chain-emar` as the wasm
compiler/linker/archiver. Those are wrapper scripts shipped by the
`clang-tool-chain` PyPI package; earlier I dropped that dep when porting
the *install* logic to Rust, but the *wrappers* are still needed at
compile time.

Two changes:

* `pyproject.toml`: re-add `clang-tool-chain>=1.2.6` to runtime deps so
  the wrapper scripts land in `.venv/Scripts/bin`. Comment explains why
  it stays after the earlier dep purge.
* `crates/fastled-cli/src/lib.rs`: after Rust installs emscripten, also
  export `CLANG_TOOL_CHAIN_DOWNLOAD_PATH` pointed at the toolchains root
  (`~/.fastled/toolchains/`). The wrappers honor this env var and will
  find the Rust-installed emscripten under
  `<root>/emscripten/<plat>/<arch>/<ver>` rather than the default
  `~/.clang-tool-chain/` layout.

Net runtime deps: 3 → 4 (`meson`, `ninja`, `clang-tool-chain`,
`zcmds_win32; Windows`). The architectural goal "Rust owns install" is
preserved — clang-tool-chain provides only the meson-compatible compiler
wrappers; the actual emscripten download still happens in Rust.

* fix(rust): restore +x on emscripten binaries after tar extract

After commit 524dac4 (clang-tool-chain dep), the deploy job advances all
the way through meson configuration — the wrappers find emscripten 4.0.21
via CLANG_TOOL_CHAIN_DOWNLOAD_PATH — and then fails the very first
compile invocation with:

    PermissionError: [Errno 13] Permission denied
    File ".../emscripten/.../bin/clang"

Some tar extractors apply the umask during unpack and drop the +x bit
from shipped binaries (clang, wasm-ld, node, etc.) under bin/,
emscripten/, libexec/. The previous Python implementation
(`emscripten_archive.py::_extract_archive`) had a belt-and-suspenders
chmod loop for exactly this; the Rust port (install.rs) dropped it.

Restore it: walk staging/{bin,emscripten,libexec} recursively after
`archive::extract_tar_zst` and add 0o111 to every file's mode. Gated on
`#[cfg(unix)]` — no-op on Windows where the umask issue doesn't apply.
zackees added a commit that referenced this pull request May 13, 2026
Sweeps the non-Python items issue #59 flagged that are still on main
after PRs #61, #62, #64, #65, #66, #68 cleaned up the Python side:

* `entrypoint.sh` — Docker container entrypoint. Sourced
  `/emsdk/emsdk_env.sh` (container path) and called
  `fastled_wasm_compiler.cli_update_from_master` (a module from a
  separate package that only exists inside the container).
* `compiler/run.py` — Docker compile-server entry point. Imports
  `fastled_wasm_compiler`, a separate package not in this repo.
* `compiler/debug.sh` — instructed user to run from `/js` (container
  working directory). Empty `compiler/` directory removed.
* `.dockerignore` — only contained cache exclusions
  (`**/.mypy_cache/`, `**/.ruff_cache/`) which belong in `.gitignore`
  if anywhere; no Docker build context this could serve.
* `chrome_vscode_bridge_design_task.md` — one-off design draft that
  opens with "Here's a refined ... file for your repo. Feel free to
  adjust...". Zero references anywhere else.
* `upload_package.sh` — manual PyPI upload helper, superseded by the
  automated `publish_release.yml` workflow. Zero inbound references.
* Empty root-level `fastled/` directory left over from a prior layout.

`lint` script updated to stop passing the (now-deleted) `compiler/` path
to ruff / black / isort / pyright / the keyboard-interrupt checker.

`bash lint` clean. `bash test` — 119 pass.

Verified by grep across `*.py`, `*.rs`, `*.toml`, `*.yml`, `*.sh`,
`*.md`, `Cargo.toml` (excluding `.venv/` and `.claude/worktrees/`):
none of the deleted files are referenced from a live code path or
workflow.
@zackees
zackees deleted the feat/bundle-rust-binary branch November 5, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bundle the Rust fastled CLI binary into the wheel; never cargo-build it in install/CI

1 participant