Skip to content

Add self-contained per-chip ESP libs slices and additive v1 selection - #1890

Merged
zackees merged 14 commits into
mainfrom
feat/esp-platform-lib-slices
Nov 4, 2026
Merged

zackees merged 14 commits into
mainfrom
feat/esp-platform-lib-slices

Conversation

@zackees

@zackees zackees commented Nov 4, 2026

Copy link
Copy Markdown
Member

Executive summary

Add deterministic self-contained ESP libs slices, additive v1 manifest assets, and MCU-specific fetching for framework-arduinoespressif32-libs. Whole bundles remain valid for old clients and every MCU. Slice integrity failures are hard errors with no whole-bundle fallback.

Measured upstream 3.3.5 once with the new slicer (decimal MB). Each slice includes its chip, hosted/, and top-level files with the upstream root prefix. Projected totals add the supplied 19.7 MB core + 168 MB toolchain + 0.48 MB esptool = 188.18 MB.

Chip Slice bytes Slice MB Projected board total MB
esp32 28,195,096 28.20 216.38
esp32c3 34,895,770 34.90 223.08
esp32c5 40,794,296 40.79 228.97
esp32c6 40,039,905 40.04 228.22
esp32h2 37,215,524 37.22 225.40
esp32p4 38,395,428 38.40 226.58
esp32p4_es 38,206,520 38.21 226.39
esp32s2 26,815,899 26.82 215.00
esp32s3 39,790,521 39.79 227.97

ESP32-S3 totals 227.97 MB, below 300 MB. This exact archive contains nine chip directories, including esp32p4_es; this table reports the observed archive rather than assuming ten chip directories. No artifacts or manifests were published; no release or consumer-pin change was made. Leave this PR unmerged, as requested.

Detail

  • ci/slice_esp_libs.py: spool reads the xz once, using numbered temporary disk payloads rather than extracting archive paths; write_slice sorts names and reconstructs ownership/time-normalized tar headers. Single-thread zstd defaults to level 19, checksum on, default window; levels above 19 are rejected. slice_archive emits the index and --check recomputes in scratch without rewriting output. Stdout includes a ready-to-paste manifest_slices map.
  • ci/gen_mirror_manifest.py: this base had no existing prebuilt ESP entry or prebuilt_documents helper. --prebuilt accepts existing whole-asset components with optional slices, emits the @esp-libs shard, and keeps v1 and all supplied whole fields intact. Publishing remains explicit; the slicer never writes site/mirror.
  • fbuild-packages-fetch/src/variants.rs: FetchRequest accepts an MCU hint only for SDK libs. Vendor URL miss uses the whole vendor archive; hit+slice selects that slice; missing chip logs and uses the mirrored whole. Relative slice URLs resolve against the manifest index base. Its successful source survives cache reloads. Existing operator mirror overrides are respected.
  • esp32_framework/libs.rs: forwards the already-resolved board MCU to the fetcher. The existing SDK completion check/merge remains MCU-specific. Reusable archives live in vendor-identity/package@version+chip directories beneath framework tools/.libs-archives; a whole archive satisfies every MCU. Preserve _es engineering-sample names. Toolchains and esptool are unchanged.
  • Tests cover self-contained two-chip output, shared top-level files (including an esp32-* filename), byte-identical reruns and hashes/sizes, absolute/parent traversal and reverse-order symlink ancestors, empty paths, good/tampered check mode, old/new manifest parsing, the selection matrix, S3/Pages bases and cached source, corrupt SHA with no bundle request, size checks, whole-cache reuse and cross-chip misses. Serde has no deny_unknown_fields.

Commands (all exit 0):

uv run --no-project --with zstandard python -m unittest ci.test_slice_esp_libs ci.test_gen_mirror_manifest ci.test_local_suites -v
uv run --no-project --with ruff ruff format --check ci/slice_esp_libs.py ci/test_slice_esp_libs.py
uv run --no-project --with ruff ruff check ci/slice_esp_libs.py ci/test_slice_esp_libs.py ci/gen_mirror_manifest.py ci/test_gen_mirror_manifest.py ci/local_gate.py ci/local_suites.py
soldr cargo test -p fbuild-packages-fetch --lib
soldr cargo clippy --workspace --all-targets -- -D warnings
bash test
curl -fL --silent --show-error https://github.com/espressif/arduino-esp32/releases/download/3.3.5/esp32-3.3.5-libs.tar.xz -o /tmp/fbuild-esp32-3.3.5-libs.tar.xz
uv run --no-project ci/slice_esp_libs.py /tmp/fbuild-esp32-3.3.5-libs.tar.xz --version 3.3.5 --out-dir /tmp/fbuild-esp-slices-measured > /tmp/fbuild-esp-slices-measured.json

27 Python tests passed. Fetch crate: 259 passed, 1 ignored. Real frame inspection: every slice checksum enabled, window 8,388,608 bytes. Source: 271,802,544 bytes, SHA-256 394170d0dbe2c728c8985129275baea61c61d56eb976873ce4cae197a12c0ed0.

Local artifacts: /tmp/fbuild-esp-slices-measured/ (slices and index) and /tmp/fbuild-esp-slices-measured.json (stdout including manifest map). Unresolved local failures: none. Hosted checks are tracked on this PR.

Refs #1889.

Recommended follow-ups

  1. Upload the reviewed slices under keys matching the printed relative URLs.
  2. Add the map to the existing whole asset and publish the v1 prebuilt shard/index.
  3. After review/merge, release fbuild and update the FastLED consumer pin; validate an end-to-end cold ESP32-S3 build.

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…dle fallback

Uploads all nine chip slices to the fbuild Spaces bucket under the v1
layout and mirrors the whole 3.3.5 bundle there too, so a manifest hit
serves the ~28-41 MB chip slice while every fallback path still has a
bucket copy. The shard carries absolute https slice URLs; validation now
accepts absolute https slice URLs while keeping the relative-path rules.
The committed-manifest test tolerates the non-registry ESP libs shard.
The ignored integration test drives FetchRequest through a served manifest
and asserts the esp32s3 slice (39.8 MB) downloads instead of the whole
271.8 MB bundle, on a chip-specific cache path.
The workspace caps integration-test binaries at eight categories. Move
slice_pilot.rs under fbuild-packages/tests/it/ as a module so it links in
the shared binary instead of adding a ninth, and rustfmt the manifest
test's registry-URL guard.
The platform manifest resolves 3.3.12 (not 3.3.5), so the pilot shard had
to cover it or clients fell back to the whole 347 MB bundle. Publish all
nine chip slices for core-3.3.12 plus the whole-bundle mirror, and add the
component keyed by the exact vendor URL pioarduino names.

Verified end to end on a clean cache: the esp32s3 build downloads the
72.3 MB slice instead of the 347 MB bundle, extracts 333 MB instead of
2.47 GB of all-chip trees, and stamps a +esp32s3 cache directory.
pioarduino 55.03.311 resolves core-3.3.11, so add its nine chip slices
plus the whole-bundle mirror to the esp-libs shard. Adds the reusable
publisher script that does download -> slice -> upload -> shard merge in
one pass.

Fixed a use-after-cleanup bug in the publisher: archive.stat() ran after
the TemporaryDirectory exited, so the whole-bundle size read always
raised FileNotFoundError.
The ESP-IDF 5.3/5.4/6.1 libs ship as ZIPs from esp32-arduino-lib-builder,
not tar.xz. The ZIP layout is identical to the tar.xz one
(esp32-arduino-libs/{chip}/...), so spool_zip normalizes ZipInfo into
TarInfo and the existing slicer works unchanged.

Also makes version_of handle ZIP filenames (v5.3-<hash>-v2.zip -> 5.3),
allows uppercase in version tokens (4.0.0-RC1), and adds --slices-dir so
already-sliced output can be uploaded without re-slicing.

Published v5.3: 7 chips, 27-46 MB each (325 MB ZIP).
spool() read 4 bytes but compared against the 5-byte '\xfd7zXZ' prefix,
so every tar.xz hit the unsupported-format branch. Read 6 bytes and
match the full xz magic.
3.0.4 (6 chips, 291 MB ZIP) and 4.0.0-RC1 (10 chips, 387 MB tar.xz) now
have per-chip slices in the esp-libs shard. Covers the remaining
pioarduino versions fastled builds: 51.03.04 -> 3.0.4, 61.04.00-RC1 ->
4.0.0-RC1.
The published-manifest contract requires every component's first URL to be
the bucket copy. The --slices-dir fast path only uploaded slices and left
urls[0] pointing at the vendor, so 5.3, 3.0.4 and core-4.0.0-RC1 violated
the contract and failed committed_manifest_is_accepted_and_prefers_the_bucket.

Download the whole bundle from the vendor URL once, upload it under
whole/, and put the bucket URL first. Also derive the whole-key suffix
from the vendor filename so ZIP releases are not mislabelled .tar.xz.

Adds docs/design/artifact-serving-decisions.md: the reviewed decisions for
CLI opt-out, key layout, manifest freshness, streaming/resume, mirror
failover, benchmarks and tests.
The repo denies std::path::PathBuf in signatures (dylint ban_std_pathbuf).
archive_path, fetch and fetch_selected returned PathBuf, so Dylint Full failed
on variants.rs. Return fbuild_core::path::NormalizedPath instead; it derefs to
Path so verification, extraction and callers are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

1 participant