Skip to content

feat(lnk): lnk pull --materialize writes resolved blobs out for non-fbuild consumers - #2110

Open
zackees wants to merge 2 commits into
mainfrom
agent/lnk-materialize
Open

zackees wants to merge 2 commits into
mainfrom
agent/lnk-materialize

Conversation

@zackees

@zackees zackees commented Nov 20, 2026

Copy link
Copy Markdown
Member

Closes no issue on its own — it is the item the #1356 audit called
"the missing consumer", and the last ruling-free piece of that issue's
materialization half.

Why

fbuild lnk pull stopped at the disk cache. The materializer itself already
existed (materialize_all / materialize_one, used by buildfs and by the
ESP32 embed_files stage), but nothing exposed it to the CLI, so the two
consumers the audit named — an SD-card copy and the fastled-wasm VFS packer —
had no supported way to get a real file. The audit's Q2 recommendation is
exactly this flag pair, and Q1 settles where the destination comes from:
the pointer names a source, so placement belongs to the consumer, never to a
dest field on the descriptor.

What

Three destinations, all opt-in; plain lnk pull is byte-for-byte unchanged:

Flag Destination
--materialize <build root>/resources/ — the projection a real build uses
--materialize-dir <DIR> an explicit root; the pointer's relative path is preserved
--in-place beside the pointer (data/a.bin.fetch → data/a.bin)
  • The pointer's directory shape is preserved in every mode, so
    data/video1.rgb.fetch never becomes a flat pile entry.
  • --in-place writes generated files into the source tree, so each name is
    recorded in the scanned root's .gitignore, idempotently — the audit's
    mitigation for the stale-file / accidental-commit risk.
  • Materialization is per-pointer best-effort: one unresolvable blob is
    reported and the others are still written (the same shape the cache-only
    summary already had). Exit status is still non-zero on any failure.
  • --materialize-dir with --in-place is refused with an error naming both.

Files: new crates/fbuild-cli/src/cli/lnk_materialize.rs (mode + target
resolution + .gitignore bookkeeping + 9 unit tests), lnk.rs dispatch,
args.rs flags, fbuild-toolchain/src/lnk/README.md and
docs/reference/cli.md.

Tests

soldr cargo test -p fbuild-cli --bin fbuild lnk — 12 passed / 0 failed
(9 new; the 3 pre-existing lnk tests still pass). New coverage: flag
combination rules, relative-path preservation, in-place target, sha-identical
materialization through the real scan → resolve → materialize path with a
cache-staged blob, per-pointer failure isolation, and .gitignore
idempotency.

soldr cargo clippy -p fbuild-cli --all-targets --no-deps -- -D warnings —
exit 0. soldr cargo fmt --check — exit 0.

Wiring verified through the real binary against a local HTTP server (not just
the helper):

$ fbuild lnk pull --materialize-dir out proj        # cache miss -> real download
file .../proj/data/clip.bin.fetch  →  out/data/clip.bin
lnk pull: materialized 1 of 1 into out              # bytes identical to the served blob
$ fbuild lnk pull --in-place proj
gitignore: added 1 generated name(s) to .../proj/.gitignore   # second run adds nothing
lnk pull: materialized 1 of 1 into beside each pointer
$ fbuild lnk pull proj
lnk pull: 1 ok, 0 failed (of 1)                     # cache-only path unchanged

args.rs is already over the 1000-LOC gate at base (baselined); the new file
is 480 lines and lnk.rs 302.

Not in this PR

The other two open cells of #1356 stay open, unchanged: the end-of-build
"fetched but unconsumed" warning (its suppression knob is a maintainer
decision) and the capability matrix as data in the board JSONs.

The materializer already existed (`materialize_all` / `materialize_one`) but
only `buildfs` and the ESP32 embed stage could reach it: `fbuild lnk pull`
stopped at the disk cache, so an SD-card user, a release script or the
fastled-wasm VFS packer had no way to get a real file.

Adds the destination to the command line instead of to the descriptor, per
the #1356 audit (Q1: git-LFS, DVC, Bazel and Nix all keep
placement out of the pointer):

* `--materialize` — the build tree's `resources/` projection (default),
* `--materialize-dir <DIR>` — an explicit root; the pointer's relative path is
  preserved (`data/a.bin.fetch` -> `<DIR>/data/a.bin`),
* `--in-place` — beside the pointer, and the generated name is recorded in the
  scanned root's `.gitignore` (idempotently), which is the audit's mitigation
  for the accidental-commit risk that in-place mode reintroduces.

One unresolvable pointer no longer hides the rest: materialization collects
per-pointer failures and reports them with the cache-only summary shape.
Plain `lnk pull` behaviour is unchanged.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

The Dylint gates run on new code, including test modules, and three path
lints fired on lnk_materialize.rs: `ban_std_pathbuf` (owned PathBuf in the
failure type, the target/root return types and the test helper),
`ban_raw_path_prefix_compare` (`Path::strip_prefix` against the scan root)
and `ban_manual_slash_normalize` (hand-rolled backslash rewrite for the
.gitignore entry).

Carry `NormalizedPath` through the module instead: destinations, targets and
recorded gitignore names are normalized values, the relative projection uses
`relative_to`, and the gitignore name uses `display_slash()`. Behaviour is
unchanged -- same three destinations, same bytes, same idempotent gitignore.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant