Repository navigation
feat(toolchain): pin the ESP-IDF Python environment as an fbuild package - #1980
Merged
Merged
Conversation
Third and last input a delegated `idf.py` build needs (#1970): the interpreter its own dependency checker insists on. - Interpreter: python-build-standalone 3.12.16 install_only, sha256-pinned per host from the release's published SHA256SUMS (68 MB on linux-amd64). A host python3 cannot be used: this one is 3.14 and half of IDF's requirements publish no cp314 wheel, so the wheel tags must come from a pinned interpreter. - Packages: data/esp-idf-v5.5.1-python.lock = the pinned commit's tools/requirements/requirements.core.txt resolved against the release's espidf.constraints.v5.5.txt with `uv pip compile --universal --generate-hashes` (60 packages, 976 hashed files). Installed with `pip install --require-hashes --no-deps`, so the lock IS the pin and one file is correct on all five hosts. - Layout: <entry>/python + <entry>/python_env/idf5.5_py3.12_env, the name idf_tools.py derives; IDF_PYTHON_ENV_PATH is exported for #1971. - `install.sh` is not involved: it dies in its venv step on this host (uv's CPython has no ensurepip) and wants a system libusb. Measured, real: venv 85,500,116 B (`du -sb`), cache entry 312,556,143 B (interpreter included). End-to-end test downloads the real archive, creates the venv, installs the lock, and runs ESP-IDF's own tools/check_python_dependencies.py against it -> "Python requirements are satisfied." (1 passed, 46.5 s), second install a no-op. 13 new unit tests; workspace clippy -D warnings clean, fmt clean, boundary 44/43, test-layout + dylint allowlists ok.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
5 tasks
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth slice of #1970 (follow-up 1 of the #1761 research pass). Parts 1-3 are on
main:tool pins (#1975,
f1ecd74b), the esp32s3-only profile +IdfEnvironment(#1976,a1e9c815), and the 450 MB source tree as 24 sha256-pinned archives (#1978,3d7e9158).This PR is the third input a delegated
idf.pybuild (#1971) needs: the Pythonenvironment
idf.pyitself runs in. Not a convenience —idf.pyrefuses to start unlesstools/check_python_dependencies.pyis satisfied by the interpreter it was launched with.Why not
install.shMeasured 2026-11-09 on this NixOS host:
install.shdies in its venv step (uv's CPythonhas no working
ensurepip) and again onopenocd(libusb-1.0.so.0). A hostpython3cannot stand in either: this host's is 3.14 and half of IDF's requirements publish no
cp314wheel. So the environment is provisioned like every other toolchain — as a pinned package.
What is pinned
3.12.16(20261026)install_onlyper host, sha256 from the release's publishedSHA256SUMSdata/esp-idf-v5.5.1-python.lock: the pinned commit'stools/requirements/requirements.core.txtresolved against the release's publishedespidf.constraints.v5.5.txtThe lock is generated with
uv pip compile --universal --generate-hashes --python-version 3.12and consumed by
pip install --require-hashes --no-deps:--require-hashesmeans the lock is the pin: any file that does not match a listedsha256 is refused,
esptool's sdist included (4.x publishes no wheel on PyPI, so pip buildsit with the setuptools that ships inside the pinned interpreter).
--universalmeans one file covers all five hosts — each line carries the hashes of everyplatform's file, so there is no per-host lock to drift.
--no-depsis deliberate: the lock is the closure; letting pip resolve could installsomething the lock never pinned.
The constraints file is committed beside the lock (
data/esp-idf-v5.5.1-constraints.txt,verbatim from the release) so the pair can be regenerated and checked offline.
Layout
<cache entry>/python(the interpreter) and<cache entry>/python_env/idf5.5_py3.12_env(the venv) — the name
idf_tools.py::get_python_env_pathderives, spelled the same so #1971 canhand the tree to
idf.pywithout a translation layer.IdfPythonEnvironment::env_vars()exportsIDF_PYTHON_ENV_PATH, which bothidf.pyandidf_tools.pyprefer overIDF_TOOLS_PATH.A venv that dies inside
pip installis not reused: the last step writesrequirements-installed.marker, andis_installed()requires it, so a partial tree is clearedand rebuilt instead of being built on.
Real output
soldr cargo test -p fbuild-toolchain --lib -- --ignored --nocapture esp_idf_python_envwith
FBUILD_IDF_SOURCE_DIRpointing at a checkout of the pinned commit:Python requirements are satisfied.is ESP-IDF's owntools/check_python_dependencies.pyoutput — the same gate
idf.pyruns at startup, not a proxy. The secondinstall()is ano-op (AC4).
Measured sizes (
du -sb): venv 85.5 MB (Espressif's own is 63 MB), cache entry312.6 MB with the interpreter. #1970's AC2 budget is tools-only (1149.2 MB), so nothing
here changes it; the release total is tools 1149.2 + source 450.0 + python env 85.5 MB.
13 new unit tests (pin table, lock integrity/coverage, layout, marker, both archive layouts,
bounded error tail). Workspace
clippy --all-targets -- -D warningsclean,fmtclean,boundary
44/43, test-layout + dylint allowlists ok.Deliberately not done
EspIdfRelease::install()yet — that is the next unit (it also givesfeat(toolchain): pin an ESP-IDF release as fbuild packages with an esp32s3-only install profile #1970 a single merged-state AC table), and it needs its own full-release e2e.
upstream
SHA256SUMS(same class astools.jsonin feat(toolchain): pin ESP-IDF v5.5.1 tool components as fbuild packages #1975) but have not been installed andchecked on that host; they are marked as such in the module doc.
Refs #1970. Blocks #1971.