Repository navigation
Add self-contained per-chip ESP libs slices and additive v1 selection - #1890
Merged
Merged
Conversation
Refs #1889. Preserve manifest v1 whole bundles, isolate chip archive caches, and fail closed on slice integrity errors.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
…dle fallback Uploads all nine chip slices to the fbuild Spaces bucket under the v1 layout and mirrors the whole 3.3.5 bundle there too, so a manifest hit serves the ~28-41 MB chip slice while every fallback path still has a bucket copy. The shard carries absolute https slice URLs; validation now accepts absolute https slice URLs while keeping the relative-path rules. The committed-manifest test tolerates the non-registry ESP libs shard.
The ignored integration test drives FetchRequest through a served manifest and asserts the esp32s3 slice (39.8 MB) downloads instead of the whole 271.8 MB bundle, on a chip-specific cache path.
The workspace caps integration-test binaries at eight categories. Move slice_pilot.rs under fbuild-packages/tests/it/ as a module so it links in the shared binary instead of adding a ninth, and rustfmt the manifest test's registry-URL guard.
The platform manifest resolves 3.3.12 (not 3.3.5), so the pilot shard had to cover it or clients fell back to the whole 347 MB bundle. Publish all nine chip slices for core-3.3.12 plus the whole-bundle mirror, and add the component keyed by the exact vendor URL pioarduino names. Verified end to end on a clean cache: the esp32s3 build downloads the 72.3 MB slice instead of the 347 MB bundle, extracts 333 MB instead of 2.47 GB of all-chip trees, and stamps a +esp32s3 cache directory.
pioarduino 55.03.311 resolves core-3.3.11, so add its nine chip slices plus the whole-bundle mirror to the esp-libs shard. Adds the reusable publisher script that does download -> slice -> upload -> shard merge in one pass. Fixed a use-after-cleanup bug in the publisher: archive.stat() ran after the TemporaryDirectory exited, so the whole-bundle size read always raised FileNotFoundError.
The ESP-IDF 5.3/5.4/6.1 libs ship as ZIPs from esp32-arduino-lib-builder,
not tar.xz. The ZIP layout is identical to the tar.xz one
(esp32-arduino-libs/{chip}/...), so spool_zip normalizes ZipInfo into
TarInfo and the existing slicer works unchanged.
Also makes version_of handle ZIP filenames (v5.3-<hash>-v2.zip -> 5.3),
allows uppercase in version tokens (4.0.0-RC1), and adds --slices-dir so
already-sliced output can be uploaded without re-slicing.
Published v5.3: 7 chips, 27-46 MB each (325 MB ZIP).
spool() read 4 bytes but compared against the 5-byte '\xfd7zXZ' prefix, so every tar.xz hit the unsupported-format branch. Read 6 bytes and match the full xz magic.
3.0.4 (6 chips, 291 MB ZIP) and 4.0.0-RC1 (10 chips, 387 MB tar.xz) now have per-chip slices in the esp-libs shard. Covers the remaining pioarduino versions fastled builds: 51.03.04 -> 3.0.4, 61.04.00-RC1 -> 4.0.0-RC1.
The published-manifest contract requires every component's first URL to be the bucket copy. The --slices-dir fast path only uploaded slices and left urls[0] pointing at the vendor, so 5.3, 3.0.4 and core-4.0.0-RC1 violated the contract and failed committed_manifest_is_accepted_and_prefers_the_bucket. Download the whole bundle from the vendor URL once, upload it under whole/, and put the bucket URL first. Also derive the whole-key suffix from the vendor filename so ZIP releases are not mislabelled .tar.xz. Adds docs/design/artifact-serving-decisions.md: the reviewed decisions for CLI opt-out, key layout, manifest freshness, streaming/resume, mirror failover, benchmarks and tests.
The repo denies std::path::PathBuf in signatures (dylint ban_std_pathbuf). archive_path, fetch and fetch_selected returned PathBuf, so Dylint Full failed on variants.rs. Return fbuild_core::path::NormalizedPath instead; it derefs to Path so verification, extraction and callers are unchanged.
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Executive summary
Add deterministic self-contained ESP libs slices, additive v1 manifest assets, and MCU-specific fetching for
framework-arduinoespressif32-libs. Whole bundles remain valid for old clients and every MCU. Slice integrity failures are hard errors with no whole-bundle fallback.Measured upstream 3.3.5 once with the new slicer (decimal MB). Each slice includes its chip,
hosted/, and top-level files with the upstream root prefix. Projected totals add the supplied 19.7 MB core + 168 MB toolchain + 0.48 MB esptool = 188.18 MB.ESP32-S3 totals 227.97 MB, below 300 MB. This exact archive contains nine chip directories, including
esp32p4_es; this table reports the observed archive rather than assuming ten chip directories. No artifacts or manifests were published; no release or consumer-pin change was made. Leave this PR unmerged, as requested.Detail
ci/slice_esp_libs.py:spoolreads the xz once, using numbered temporary disk payloads rather than extracting archive paths;write_slicesorts names and reconstructs ownership/time-normalized tar headers. Single-thread zstd defaults to level 19, checksum on, default window; levels above 19 are rejected.slice_archiveemits the index and--checkrecomputes in scratch without rewriting output. Stdout includes a ready-to-pastemanifest_slicesmap.ci/gen_mirror_manifest.py: this base had no existing prebuilt ESP entry orprebuilt_documentshelper.--prebuiltaccepts existing whole-asset components with optional slices, emits the@esp-libsshard, and keeps v1 and all supplied whole fields intact. Publishing remains explicit; the slicer never writessite/mirror.fbuild-packages-fetch/src/variants.rs:FetchRequestaccepts an MCU hint only for SDK libs. Vendor URL miss uses the whole vendor archive; hit+slice selects that slice; missing chip logs and uses the mirrored whole. Relative slice URLs resolve against the manifest index base. Its successful source survives cache reloads. Existing operator mirror overrides are respected.esp32_framework/libs.rs: forwards the already-resolved board MCU to the fetcher. The existing SDK completion check/merge remains MCU-specific. Reusable archives live in vendor-identity/package@version+chip directories beneath frameworktools/.libs-archives; a whole archive satisfies every MCU. Preserve_esengineering-sample names. Toolchains and esptool are unchanged.esp32-*filename), byte-identical reruns and hashes/sizes, absolute/parent traversal and reverse-order symlink ancestors, empty paths, good/tampered check mode, old/new manifest parsing, the selection matrix, S3/Pages bases and cached source, corrupt SHA with no bundle request, size checks, whole-cache reuse and cross-chip misses. Serde has nodeny_unknown_fields.Commands (all exit 0):
27 Python tests passed. Fetch crate: 259 passed, 1 ignored. Real frame inspection: every slice checksum enabled, window 8,388,608 bytes. Source: 271,802,544 bytes, SHA-256
394170d0dbe2c728c8985129275baea61c61d56eb976873ce4cae197a12c0ed0.Local artifacts:
/tmp/fbuild-esp-slices-measured/(slices and index) and/tmp/fbuild-esp-slices-measured.json(stdout including manifest map). Unresolved local failures: none. Hosted checks are tracked on this PR.Refs #1889.
Recommended follow-ups