Repository navigation
perf(espidf): declare delegated IDF builds cold-only, with the measurement behind it (#1974) - #1987
Merged
Merged
Conversation
A delegated `idf.py` build hands compilation to ninja, so fbuild's embedded zccache dispatcher never sees its compiler invocations. #1974 has to be a decision with a number, not a surprise, so the wrapper option was measured before being rejected: cold, no launcher 41.8 s no-op (ninja up to date) 1.2 s fullclean + build, no launcher 34.0 s cold, zccache launcher 50.6 s (233 MB stored) new build dir, launcher, warm 67.4 s no reuse (436 MB) fullclean + build, launcher 110.6 s no reuse (640 MB) zccache reports a 0.2% hit rate over 2838 compilations: the key covers ninja's command line, which embeds the absolute build directory (-o, -MF, -I, -fmacro-prefix-map), so every build directory and every fullclean recompiles everything at 1.2-3.3x the unwrapped wall time. Inside one build directory the only rebuild it could serve is one ninja already answers in 1.2 s. So the delegated path is declared cold-only, in the report and not just in a comment: - `IdfCacheMode` (ColdOnly) with a `label()` and a build-log line, so the mode a summary or index names is the mode that ran; - `esp32_idf_cmake_stack_summary` carries `cache=cold-only` beside `integration=idf-cmake`; - `idf-artifacts.json` records `"cache": "cold-only"`. Tests pin both halves: the summary cannot read as a cache hit, and the artifact index has to name the mode. Design note gets the measurement table.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up 5 of 5 of the #1761 research pass: caching for delegated ESP-IDF builds. Closes #1974.
Decision: cold-only, declared (not "maybe later")
The wrapper option was measured before being rejected. Harness:
fbuild-steward/probes/idf_cache_bench.py(local, untracked) replays the exactchild environment the delegated build builds (
IDF_PATH,IDF_PYTHON_ENV_PATH,IDF_TOOLS_PATH, profilePATHprepended to the hostPATH) and passes thelauncher as
-DCMAKE_C_COMPILER_LAUNCHER=… -DCMAKE_CXX_COMPILER_LAUNCHER=…—the only insertion point, because ninja spawns gcc. Pinned release v5.5.1
(
fcae3288), spike projecttests/platform/espidf_component, 1047 ninjacompile actions, host load average 9–30 (the deltas are far outside that noise).
ninjaup to date)fullclean+ build, no launcherzccachelauncher, empty cachefullclean+ build, launcherzccache --show-statsafter those runs: 2838 compilations, 0.2 % hit rate.The wrapper works (replaying one identical entry does hit); what it cannot reuse
is ninja's command line, which embeds the absolute build directory (
-o,-MF,-I,-fmacro-prefix-map). Every build directory and everyfullcleanthereforerecompiles everything, at 1.2–3.3× the unwrapped wall time; inside one build
directory the only rebuild it could serve is the one ninja answers in 1.2 s.
A future launcher must key on IDF release +
sdkconfigdigest rather than the rawcommand line before it can claim anything.
What the build now says
IdfCacheMode::ColdOnlywithlabel()andlog_line()— a type, so a reportcannot quietly claim a cache the build did not have.
…, integration=idf-cmake, cache=cold-only, idf=v5.5.1; proof: …(also fixes the doubled spacing that string had).
idf.py caching: cold-only — compiler invocations are driven by ninja, so no object is served from fbuild's zccache; ….<build>/idf/idf-artifacts.json:"cache": "cold-only".Tests (real output)
soldr cargo test -p fbuild-build-esp --lib→ 244 passed; 0 failed; 2 ignored(up from 242:
the_delegated_build_reports_cold_only_cachingandidf_cmake_summary_claims_cold_only_caching_and_no_cache_hit, which asserts thesummary says
cache=cold-onlyand contains no "cached"/"cache=hit" claim).idf-artifacts.jsontest assertsindex["cache"] == "cold-only".espidf_component_project_builds_and_publishes_artifacts→ 1 passed in 48.4 s (warm install, full cold delegated build through the orchestrator).
soldr cargo clippy --workspace --all-targets -- -D warnings→ exit 0.soldr cargo fmt --all -- --check→ 0 diffs;ci/check_test_layout.pyok;ci/enforce_platform_boundary.py --print-totals→rows=44; dylint_rows=43.Docs
docs/design/bare-idf-component-builds.mdgets the measurement table (question 3and follow-up 5 is marked decided.
Non-goals respected: no change to the Arduino-on-IDF caching path, no new CI lane.