Skip to content

feat(toolchain): pin the ESP-IDF Python environment as an fbuild package - #1980

Merged
zackees merged 1 commit into
mainfrom
agent/esp-idf-python-env
Nov 10, 2026
Merged

zackees merged 1 commit into
mainfrom
agent/esp-idf-python-env

Conversation

@zackees

@zackees zackees commented Nov 10, 2026

Copy link
Copy Markdown
Member

Fourth slice of #1970 (follow-up 1 of the #1761 research pass). Parts 1-3 are on main:
tool pins (#1975, f1ecd74b), the esp32s3-only profile + IdfEnvironment (#1976,
a1e9c815), and the 450 MB source tree as 24 sha256-pinned archives (#1978, 3d7e9158).

This PR is the third input a delegated idf.py build (#1971) needs: the Python
environment idf.py itself runs in. Not a convenience — idf.py refuses to start unless
tools/check_python_dependencies.py is satisfied by the interpreter it was launched with.

Why not install.sh

Measured 2026-11-09 on this NixOS host: install.sh dies in its venv step (uv's CPython
has no working ensurepip) and again on openocd (libusb-1.0.so.0). A host python3
cannot stand in either: this host's is 3.14 and half of IDF's requirements publish no cp314
wheel. So the environment is provisioned like every other toolchain — as a pinned package.

What is pinned

Input Pin Size
Interpreter python-build-standalone 3.12.16 (20261026) install_only per host, sha256 from the release's published SHA256SUMS 67.9 MB (linux-amd64)
Packages data/esp-idf-v5.5.1-python.lock: the pinned commit's tools/requirements/requirements.core.txt resolved against the release's published espidf.constraints.v5.5.txt 88 KB / 60 packages / 976 hashed files

The lock is generated with uv pip compile --universal --generate-hashes --python-version 3.12
and consumed by pip install --require-hashes --no-deps:

  • --require-hashes means the lock is the pin: any file that does not match a listed
    sha256 is refused, esptool's sdist included (4.x publishes no wheel on PyPI, so pip builds
    it with the setuptools that ships inside the pinned interpreter).
  • --universal means one file covers all five hosts — each line carries the hashes of every
    platform's file, so there is no per-host lock to drift.
  • --no-deps is deliberate: the lock is the closure; letting pip resolve could install
    something the lock never pinned.

The constraints file is committed beside the lock (data/esp-idf-v5.5.1-constraints.txt,
verbatim from the release) so the pair can be regenerated and checked offline.

Layout

<cache entry>/python (the interpreter) and <cache entry>/python_env/idf5.5_py3.12_env
(the venv) — the name idf_tools.py::get_python_env_path derives, spelled the same so #1971 can
hand the tree to idf.py without a translation layer. IdfPythonEnvironment::env_vars() exports
IDF_PYTHON_ENV_PATH, which both idf.py and idf_tools.py prefer over IDF_TOOLS_PATH.

A venv that dies inside pip install is not reused: the last step writes
requirements-installed.marker, and is_installed() requires it, so a partial tree is cleared
and rebuilt instead of being built on.

Real output

soldr cargo test -p fbuild-toolchain --lib -- --ignored --nocapture esp_idf_python_env
with FBUILD_IDF_SOURCE_DIR pointing at a checkout of the pinned commit:

Python requirements are satisfied.
venv 85500116 B, entry 312556143 B
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 197 filtered out; finished in 46.53s

Python requirements are satisfied. is ESP-IDF's own tools/check_python_dependencies.py
output — the same gate idf.py runs at startup, not a proxy. The second install() is a
no-op (AC4).

Measured sizes (du -sb): venv 85.5 MB (Espressif's own is 63 MB), cache entry
312.6 MB with the interpreter. #1970's AC2 budget is tools-only (1149.2 MB), so nothing
here changes it; the release total is tools 1149.2 + source 450.0 + python env 85.5 MB.

13 new unit tests (pin table, lock integrity/coverage, layout, marker, both archive layouts,
bounded error tail). Workspace clippy --all-targets -- -D warnings clean, fmt clean,
boundary 44/43, test-layout + dylint allowlists ok.

Deliberately not done

Refs #1970. Blocks #1971.

Third and last input a delegated `idf.py` build needs (#1970): the
interpreter its own dependency checker insists on.

- Interpreter: python-build-standalone 3.12.16 install_only, sha256-pinned
  per host from the release's published SHA256SUMS (68 MB on linux-amd64).
  A host python3 cannot be used: this one is 3.14 and half of IDF's
  requirements publish no cp314 wheel, so the wheel tags must come from a
  pinned interpreter.
- Packages: data/esp-idf-v5.5.1-python.lock = the pinned commit's
  tools/requirements/requirements.core.txt resolved against the release's
  espidf.constraints.v5.5.txt with `uv pip compile --universal
  --generate-hashes` (60 packages, 976 hashed files). Installed with
  `pip install --require-hashes --no-deps`, so the lock IS the pin and one
  file is correct on all five hosts.
- Layout: <entry>/python + <entry>/python_env/idf5.5_py3.12_env, the name
  idf_tools.py derives; IDF_PYTHON_ENV_PATH is exported for #1971.
- `install.sh` is not involved: it dies in its venv step on this host
  (uv's CPython has no ensurepip) and wants a system libusb.

Measured, real: venv 85,500,116 B (`du -sb`), cache entry 312,556,143 B
(interpreter included). End-to-end test downloads the real archive, creates
the venv, installs the lock, and runs ESP-IDF's own
tools/check_python_dependencies.py against it -> "Python requirements are
satisfied." (1 passed, 46.5 s), second install a no-op.

13 new unit tests; workspace clippy -D warnings clean, fmt clean,
boundary 44/43, test-layout + dylint allowlists ok.
@coderabbitai

coderabbitai Bot commented Nov 10, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

1 participant