Skip to content

feat(espidf): build an opted-in component project with the pinned IDF - #1984

Merged
zackees merged 3 commits into
mainfrom
agent/esp-idf-delegate
Nov 10, 2026
Merged

zackees merged 3 commits into
mainfrom
agent/esp-idf-delegate

Conversation

@zackees

@zackees zackees commented Nov 10, 2026

Copy link
Copy Markdown
Member

Why

fbuild compiles every ESP32 environment with the Arduino core against
precompiled ESP-IDF .a libraries: there is no CMake and no idf.py, so a
genuine idf_component_register() project cannot be built at all (#1664), and
since #1730 it is rejected instead of being silently mis-built. This is
follow-up 2 of the research programme in #1761 (docs/design/bare-idf-component-builds.md,
recommended option 3): keep the native Arduino path untouched and add an
explicit opt-in delegating path.

Depends on #1970, whose whole release (source, toolchains, CMake, Ninja, Python
environment) is already on main.

What

Opt-in. board_build.build_system = espidf (default native, so no
existing project changes behaviour). An unknown value is a config error naming
the valid ones. With the opt-in, a bare framework = espidf is permitted; the
#1730 rejection and its warning for arduino, espidf are unchanged without it.
The chip must be esp32s3 — the one target the pinned release provisions — and
another MCU is a configuration error by name, not a missing compiler later.

Delegation (esp32/espidf_delegate.rs). After the common BuildContext
(so the board and build directory are resolved) the orchestrator branches
before any Arduino package/framework work:

<venv>/bin/python $IDF_PATH/tools/idf.py -C <project> -B <build>/idf --no-hints set-target esp32s3
<venv>/bin/python $IDF_PATH/tools/idf.py -C <project> -B <build>/idf --no-hints build

PATH is the profile's entries prepended to the host PATH and
IDF_PATH/IDF_TOOLS_PATH point into the fbuild cache tree, both measured
requirements from #1983.

Artifacts (esp32/espidf_artifacts.rs). Discovery reads IDF's own
flasher_args.json rather than hard-coding IDF's layout, validates every file,
publishes the two nested ones flat, and writes idf-artifacts.json (name, path,
byte count) as the artifact proof follow-ups 3–5 consume. All of
bin/elf/map/bootloader.bin/partition-table.bin/sdkconfig/
compile_commands.json end up under <build>/idf/.

Structured failures. A new additive core API,
fbuild_core::subprocess::run_command_reporting, spawns with a deadline but
keeps both streams: run_command drops the output of a timed-out child with
the cancelled capture future, so "the build timed out, here is what it printed"
was not expressible. The streams are drained into shared buffers (so a
surviving grandchild costs the last bytes, never the report), bounded to a
64 KiB tail with the total byte count recorded, and the tree is stopped with the
existing containment. DelegateFailure renders stage, argv, cwd, exit code,
budget and both bounded streams into the build error; a timeout is
FbuildError::Timeout with the same body.

Non-goals (per the issue): no zccache integration, no compile-db re-rooting, no
multi-version IDF matrix, and no new integration-mode string — that lands with
the coverage gate in #1972.

Real output

E2E (ignored test, real install + real idf.py over tests/platform/espidf_component,
a main component that REQUIRES mycomp plus a second component):

E2E (ignored test, real install + real idf.py over
tests/platform/espidf_component: a main component that REQUIRES mycomp
plus a second component):

soldr cargo test -p fbuild-build-esp --lib -- --ignored espidf_component_project
→ test result: ok. 1 passed; 0 failed; 236 filtered out; finished in 129.30s
(warm install). The first, cold run of the same build took 311 s including the
~1.8 GB release install.

Published under <build>/idf/, real byte counts from idf-artifacts.json:

artifact bytes
component_demo.bin 208,528
component_demo.elf 3,915,304
component_demo.map 3,191,030
bootloader.bin 21,056
partition-table.bin 3,072
sdkconfig 73,024
compile_commands.json 15,537,131
total 22,949,145

The build directory is 163 MB, the compile database has more than 100 entries,
and the app image (208,528 B) and sdkconfig (73,024 B) match the #1761
reference spike byte for byte.

One defect only a real build could find: idf.py writes the generated
sdkconfig in the project directory, next to sdkconfig.defaults, not in
the build directory, so the first artifact pass failed with "did not produce
the generated sdkconfig". generated_sdkconfig() now reads
<project>/sdkconfig (with the build-directory copy as a fallback) and
publishes it flat.

Unit gates: fbuild-core --lib 382 passed / 2 ignored, fbuild-build-esp --lib 235 passed / 2 ignored, workspace clippy --all-targets -- -D warnings
clean, cargo fmt --check clean.

Acceptance criteria

  • A real component project builds through fbuild and produces all artifacts — see above.
  • Without the opt-in the same project still fails fast with fix(esp32): reject framework = espidf instead of silently building Arduino (#1664) #1730's message (the_delegated_opt_in_permits_a_bare_espidf_project, bare_espidf_framework_is_rejected_before_building).
  • Arduino path byte-identical: no native-path code was edited; the branch is guarded by a switch that defaults to native. Not re-measured on this host (loadavg 31/16, full pioarduino core build) — CI's existing ESP32 jobs are the measurement.
  • A killed/timed-out delegated build produces a structured failure with both bounded streams and no leftover children (a_wedged_step_times_out_with_both_streams, timeout_stops_the_child_and_keeps_what_it_printed).
  • Tests exercise the real spawn against a fake idf.py (argv, cwd, env, artifact discovery) plus one ignored end-to-end test.

Closes #1971. Unblocks #1972, #1973, #1974.

Relationship to other work

Distinct from draft #1878 (which vendors IDF through platformio/framework-espidf
packages for C5/P4): this path provisions one pinned release in fbuild's own
cache and delegates the project's own CMake build, which is what #1971 scoped.

fbuild had no CMake and no idf.py, so a genuine idf_component_register()
project could not be built at all (#1664) and has been rejected
since #1730. Add the explicit opt-in that changes that, and only that:
board_build.build_system = espidf (default `native`, so no existing project
changes behaviour).

The orchestrator branches right after the common BuildContext, so an opted-in
environment never resolves the Arduino core, its archives or its linker
recipe. Two supervised steps run through fbuild_core::subprocess:  idf.py
set-target <chip>, then idf.py build, both with -C <project> -B <build>/idf.

Artifacts are discovered from IDF's own flasher_args.json rather than a second
copy of IDF's layout, validated, published flat under <build>/idf/ and recorded
in idf-artifacts.json as the proof follow-ups 3-5 consume. The generated
sdkconfig lives in the project directory, not the build directory - found only
by running the real build.

A new additive core API, run_command_reporting, keeps both bounded streams when
a deadline fires; run_command drops them with the cancelled capture future.
DelegateFailure renders stage, argv, cwd, exit code, budget and both stream
tails into the build error, so a delegated failure is never a bare exit code.

Real e2e (ignored, tests/platform/espidf_component): 1 passed in 129.30s warm,
311 s cold with the release install; app image 208,528 B and sdkconfig 73,024 B
match the #1761 spike byte for byte, 22,949,145 B of artifacts published.

Closes #1971.
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

The delegated-build branch pushed build.rs from 999 to 1017 lines, which the
"Reject .rs files over 1000 LOC" job fails on a file that was legal at the
base commit. Move the two guards into prepare_build_system (they belong beside
reject_unsupported_esp_framework) and the disk pre-flight's guard + phase into
disk_requirements::preflight_if_needed, which owns those numbers. Behaviour is
unchanged: both run at exactly the same points, before package resolution.

build.rs is 996 lines.
Dylint Full rejected two spellings in the new tests, both already recorded
as pitfalls:
- ban_std_pathbuf on a Vec<std::path::PathBuf> built from split_paths in the
  argv test; the entries are NormalizedPath now.
- ban_raw_path_prefix_compare on Path::strip_prefix in the fixture copy;
  NormalizedPath::relative_to decides containment instead.

235 lib tests still pass.
@zackees
zackees added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit ed2c845 Nov 10, 2026
38 checks passed
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

feat(espidf): opt-in delegating idf.py build producing <build>/idf/ artifacts

1 participant