Skip to content

[codex] publish fbuild usb vidpid overlay - #907

Merged
zackees merged 4 commits into
mainfrom
codex/usb-vid-proto-online-data
Jul 1, 2026
Merged

zackees merged 4 commits into
mainfrom
codex/usb-vid-proto-online-data

Conversation

@zackees

@zackees zackees commented Jul 1, 2026 •

Copy link
Copy Markdown
Member

What changed

  • Add online-data-tools/build_usb_vid_proto.py to encode merged usb-vid.json into the usb-vids.proto.zstd protobuf/zstd runtime artifact that fbuild_core::usb already knows how to read.
  • Wire Update data to publish data/usb-vids.proto.zstd on the online-data branch whenever the USB merge succeeds.
  • Move USB_VIDS_PROTO_ZSTD_URL from fastled.github.io/boards to the fbuild online-data branch.
  • Surface CDC classification through fbuild port scan, daemon DeviceState, device list/status API responses, and CLI fbuild device output.
  • Document source precedence: third-party SDK/package rows are weak supplemental inputs that fill gaps only; local FastLED board data and first-party/vendor sources remain stronger.

Why

The runtime scan path was fetching a compact VID/PID overlay from fastled.github.io/boards, while this repo's online-data workflow only published JSON/SQLite/vendor artifacts. That left fbuild dependent on an external board artifact for product-level VID/PID resolution. This makes fbuild publish and consume its own compact artifact.

The device surfaces also exposed vendor/product but not the requested CDC-vs-bridge signal. The new nullable is_cdc field reports true for CDC ACM, false for USB serial bridge, and null when the host cannot classify the port.

Follow-up note

First-party repo board data remains the support boundary: if a board is not present under crates/fbuild-config/assets/boards, it may not exist as an fbuild-supported board. Remaining product-PID gaps should be researched from internet/vendor sources only for boards that are present in that repo board set; third-party SDK/package findings can weakly fill missing VID/PID names but must not override first-party, vendor-owned, or generic USB-ID winners.

Validation

  • uv run --no-project --with pytest --with zstandard pytest online-data-tools/test_usb_vid_proto.py -q
  • soldr cargo test -p fbuild-core usb::data --lib
  • soldr cargo test -p fbuild-cli port_scan --lib
  • soldr cargo test -p fbuild-cli cdc_suffix --lib
  • soldr cargo test -p fbuild-daemon device_manager --lib
  • soldr cargo test -p fbuild-daemon deploy_port --lib
  • soldr cargo test -p fbuild-cli --lib
  • soldr cargo test -p fbuild-daemon --lib
  • soldr cargo test -p fbuild-serial --lib
  • soldr cargo test -p fbuild-packages --lib
  • soldr cargo clippy -p fbuild-serial --all-targets -- -D warnings
  • soldr cargo clippy -p fbuild-build --tests -- -D warnings
  • soldr cargo doc -p fbuild-build --no-deps with RUSTDOCFLAGS=-D warnings
  • soldr cargo doc -p fbuild-cli --no-deps with RUSTDOCFLAGS=-D warnings
  • docker run --rm -v "${PWD}:/work" -w /work rust:1.94.1 bash -lc "export PATH=/usr/local/cargo/bin:$PATH; cargo fmt --all -- --check"
  • uv run --no-project python ci/render_workflows.py --check
  • uv run --no-project python -m py_compile ci/validate_boards.py
  • git diff --check

Also generated the artifact from the current published usb-vid.json: 1,946 vendors, 22,258 products, 260,305-byte usb-vids.proto.zstd.

Summary by CodeRabbit

  • New Features

    • Added richer device and port details, including CDC/USB classification in device listings and status output.
    • Expanded online data refreshes to publish a compact USB VID:PID overlay for faster startup and better device matching.
  • Bug Fixes

    • Improved line-count checks so existing oversized files no longer block changes unless the PR introduces new violations.
    • Fixed several path-handling and normalization issues for more reliable behavior across platforms.

@coderabbitai

coderabbitai Bot commented Jul 1, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d034e93c-36bc-4bc4-8da7-daa128e99838

📥 Commits

Reviewing files that changed from the base of the PR and between f43acc3 and ab58ace.

📒 Files selected for processing (99)
  • .github/workflows/dylint.yml
  • .github/workflows/loc-gate.yml
  • .github/workflows/template_build.yml
  • .github/workflows/update-data.yml
  • ci/validate_boards.py
  • crates/fbuild-build/src/apollo3/orchestrator.rs
  • crates/fbuild-build/src/build_info.rs
  • crates/fbuild-build/src/ch32v/ch32v_compiler.rs
  • crates/fbuild-build/src/ch32v/orchestrator.rs
  • crates/fbuild-build/src/compile_backend.rs
  • crates/fbuild-build/src/compiler.rs
  • crates/fbuild-build/src/compiler_tests.rs
  • crates/fbuild-build/src/pipeline/library.rs
  • crates/fbuild-build/src/renesas/renesas_compiler.rs
  • crates/fbuild-build/src/sam/orchestrator.rs
  • crates/fbuild-build/src/source_scanner.rs
  • crates/fbuild-build/src/symbol_analyzer/mod.rs
  • crates/fbuild-build/src/zccache_embedded.rs
  • crates/fbuild-build/tests/stm32_acceptance.rs
  • crates/fbuild-build/tests/teensy30_acceptance.rs
  • crates/fbuild-build/tests/teensylc_acceptance.rs
  • crates/fbuild-cli/src/cli/build.rs
  • crates/fbuild-cli/src/cli/clang_tools.rs
  • crates/fbuild-cli/src/cli/clangd_config.rs
  • crates/fbuild-cli/src/cli/compile_many.rs
  • crates/fbuild-cli/src/cli/daemon_cmd.rs
  • crates/fbuild-cli/src/cli/device.rs
  • crates/fbuild-cli/src/cli/dispatch.rs
  • crates/fbuild-cli/src/cli/port_scan.rs
  • crates/fbuild-cli/src/cli/purge.rs
  • crates/fbuild-cli/src/cli/serial_probe.rs
  • crates/fbuild-cli/src/cli/show.rs
  • crates/fbuild-cli/src/cli/sync_cmd.rs
  • crates/fbuild-cli/src/cli/tests.rs
  • crates/fbuild-cli/src/daemon_client/types.rs
  • crates/fbuild-cli/src/sync/lockfile.rs
  • crates/fbuild-cli/src/sync/mod.rs
  • crates/fbuild-cli/src/sync/source.rs
  • crates/fbuild-cli/src/update_check.rs
  • crates/fbuild-config/Cargo.toml
  • crates/fbuild-config/src/bin/enrich_boards.rs
  • crates/fbuild-config/src/board/tests.rs
  • crates/fbuild-config/src/board/tests_enriched_json.rs
  • crates/fbuild-config/src/ini_parser/tests.rs
  • crates/fbuild-config/src/sdkconfig.rs
  • crates/fbuild-core/src/fs.rs
  • crates/fbuild-core/src/path.rs
  • crates/fbuild-core/src/subprocess.rs
  • crates/fbuild-core/src/usb/data.rs
  • crates/fbuild-core/src/usb/mod.rs
  • crates/fbuild-daemon/src/context.rs
  • crates/fbuild-daemon/src/device_manager.rs
  • crates/fbuild-daemon/src/device_manager/tests.rs
  • crates/fbuild-daemon/src/handlers/devices.rs
  • crates/fbuild-daemon/src/handlers/emulator/select.rs
  • crates/fbuild-daemon/src/handlers/operations/build.rs
  • crates/fbuild-daemon/src/handlers/operations/deploy.rs
  • crates/fbuild-daemon/src/handlers/operations/deploy_port.rs
  • crates/fbuild-daemon/src/handlers/websockets.rs
  • crates/fbuild-daemon/src/main.rs
  • crates/fbuild-daemon/src/models.rs
  • crates/fbuild-deploy/src/teensy/usb_type.rs
  • crates/fbuild-header-scan/Cargo.toml
  • crates/fbuild-header-scan/src/walker.rs
  • crates/fbuild-library-select/Cargo.toml
  • crates/fbuild-library-select/src/cache.rs
  • crates/fbuild-library-select/src/lib.rs
  • crates/fbuild-packages/src/cache.rs
  • crates/fbuild-packages/src/downloader.rs
  • crates/fbuild-packages/src/lib.rs
  • crates/fbuild-packages/src/library/ch32v_core.rs
  • crates/fbuild-packages/src/library/library_manager.rs
  • crates/fbuild-packages/src/lnk/resolver.rs
  • crates/fbuild-packages/src/toolchain/esp32_metadata.rs
  • crates/fbuild-packages/tests/lnk_e2e.rs
  • crates/fbuild-paths/src/lib.rs
  • crates/fbuild-paths/src/running_process.rs
  • crates/fbuild-python/src/daemon.rs
  • crates/fbuild-serial/src/boards.rs
  • crates/fbuild-serial/src/port_class.rs
  • crates/fbuild-serial/src/preemption.rs
  • crates/fbuild-test-support/Cargo.toml
  • crates/fbuild-test-support/src/elf_probe.rs
  • crates/fbuild-test-support/src/mini_framework.rs
  • docs/online-data.md
  • dylints/ban_env_var_set_after_import/src/lib.rs
  • dylints/ban_poison_panic/src/lib.rs
  • dylints/ban_runtime_new_outside_main/src/lib.rs
  • dylints/ban_std_fs_in_async/src/allowlist.txt
  • dylints/ban_std_sync_mutex_in_async/src/lib.rs
  • dylints/ban_unrooted_tempdir/src/lib.rs
  • dylints/ban_unwrap_in_production/src/lib.rs
  • dylints/require_multi_thread_flavor_when_spawning/src/lib.rs
  • dylints/require_oncelock_install_before_use/src/lib.rs
  • online-data-tools/README.md
  • online-data-tools/build_usb_vid_proto.py
  • online-data-tools/extract_fastled_board_usb_pids.py
  • online-data-tools/test_fastled_board_usb_pids.py
  • online-data-tools/test_usb_vid_proto.py

📝 Walkthrough

Walkthrough

This PR adds a compact USB VID:PID protobuf/zstd overlay with online-cache fetch/populate logic and CDC-ACM detection surfaced through the daemon device model, CLI device/port-scan commands, and new build/extraction Python tooling. It also migrates path canonicalization to async NormalizedPath helpers, refactors dylint test-module detection, consolidates test tempdir creation, pins a CH32V core commit, fixes CH32V variant/series mapping, updates CI workflows (dylint retries, LOC-gate baselining, board overrides), hardens several mutex-poisoning paths, and applies numerous formatting-only cleanups.

Changes

USB VID:PID Overlay and CDC Detection

Layer / File(s) Summary
Core online overlay data & cache population
crates/fbuild-core/src/usb/data.rs, crates/fbuild-core/src/usb/mod.rs
Adds try_install/populate_online_cache_from_paths(_and_urls) functions, TTL freshness checks, fetch-to-temp-then-rename logic, and expanded public re-exports.
Daemon device manager CDC classification
crates/fbuild-daemon/src/device_manager.rs, .../device_manager/tests.rs
DeviceState/DiscoveredDevice gain is_cdc; detect_is_cdc computes it during refresh and it's propagated through port moves/updates.
Daemon API and CLI models propagate is_cdc
crates/fbuild-daemon/src/handlers/devices.rs, .../models.rs, crates/fbuild-cli/src/daemon_client/types.rs, .../handlers/operations/deploy_port.rs
Handler responses and client-facing models add is_cdc: Option<bool> fields.
fbuild device CLI CDC display
crates/fbuild-cli/src/cli/device.rs
Adds with_cdc_suffix/cdc_label helpers and wires CDC labeling into List/Status output with unit tests.
fbuild port-scan overlay fetch and CDC rendering
crates/fbuild-cli/src/cli/port_scan.rs
Switches to online-data tier-2 overlay with separate proto/json cache paths, adds cdc= labeling, updates tests for fallback and CDC classification.
Daemon startup overlay population and async fs
crates/fbuild-daemon/src/main.rs
Spawns populate_usb_overlay_best_effort on startup and converts PID/port file lifecycle to async fs operations.
Overlay generation tooling, CI workflow, and docs
online-data-tools/build_usb_vid_proto.py, online-data-tools/extract_fastled_board_usb_pids.py, online-data-tools/test_*.py, online-data-tools/README.md, .github/workflows/update-data.yml, docs/online-data.md
New scripts generate the compressed protobuf overlay and extract FastLED board USB PIDs; workflow and docs are updated to publish/document the artifact.

Estimated code review effort: 4 (Complex) | ~60 minutes

Path Normalization Migration

Layer / File(s) Summary
Core fs/path helpers
crates/fbuild-core/src/fs.rs, .../path.rs
write_atomic/write_atomic_sync build fallback temp paths via NormalizedPath.
Compiler framework-source normalization
crates/fbuild-build/src/ch32v/ch32v_compiler.rs, .../renesas/renesas_compiler.rs, .../pipeline/library.rs
Switches framework-source and library-root dedup checks from canonicalize+Windows-strip to NormalizedPath.
CLI async normalize_path adoption
crates/fbuild-cli/src/cli/build.rs, .../clang_tools.rs, .../clangd_config.rs, .../compile_many.rs, .../dispatch.rs, .../tests.rs
normalize_path/build_ci_pio_env become async, using canonicalize_existing.
Sync module standardizes on NormalizedPath
crates/fbuild-cli/src/sync/mod.rs, .../sync_cmd.rs
SyncArgs, SyncOutcome::Wrote, and SyncError::NoPlatformioIni switch payload types.
update_check cache path and canonicalization tests
crates/fbuild-cli/src/update_check.rs, .../compiler_tests.rs, .../mini_framework.rs, .../cache.rs
Cache path helper returns NormalizedPath; tests adopt async canonicalization.

Estimated code review effort: 3 (Moderate) | ~30 minutes

CI Workflow and Board Validation Infra

Layer / File(s) Summary
Dylint retry and LOC-gate baseline logic
.github/workflows/dylint.yml, .../loc-gate.yml, .../template_build.yml
Computes retry attempts dynamically, compares LOC against base commit, adds a free-space guard env var.
Board build-field override validation
ci/validate_boards.py
Adds FBUILD_BUILD_FIELD_OVERRIDES for per-board expected-build corrections.

Estimated code review effort: 2 (Simple) | ~15 minutes

Test Temp-Directory Consolidation, Dylint Refactor, CH32V Fixes, and Resilience/Formatting Cleanups

Layer / File(s) Summary
Shared temp_subdir-based test helpers
multiple crates' test files
Consolidates TempDir::new()/std::env::temp_dir() usage into shared tempdir()/named_temp_file() helpers.
Dylint test-scope detection refactor
dylints/*/src/lib.rs
Refactors cfg(test)/test-module detection across multiple lints to a shared iterator-based approach.
CH32V variant/series mapping and version pin
crates/fbuild-packages/src/library/ch32v_core.rs, crates/fbuild-build/src/ch32v/orchestrator.rs
Pins a new core commit and fixes broken-variant fallback plus CH32L series mapping.
Poisoned-lock recovery
crates/fbuild-serial/src/preemption.rs, crates/fbuild-packages/src/downloader.rs, .../lib.rs
Recovers from poisoned mutex guards instead of panicking.
Formatting-only reflows
various files across build/daemon/CLI/core/serial
Non-functional reformatting of match arms, closures, and doc comments.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Daemon as fbuild-daemon
  participant PortScan as fbuild port scan CLI
  participant UsbData as fbuild_core::usb::data
  participant Remote as online-data HTTP source
  participant Cache as Local proto/json cache

  Daemon->>UsbData: populate_online_cache_from_paths(proto, json)
  UsbData->>Remote: fetch usb-vids.proto.zstd
  alt proto available
    Remote-->>UsbData: 200 OK bytes
    UsbData->>Cache: write .tmp then rename
    UsbData->>UsbData: try_install_online_cache_proto_zstd
  else proto missing/404
    UsbData->>Remote: fetch usb-vid.json
    Remote-->>UsbData: 200 OK bytes
    UsbData->>Cache: write .tmp then rename
    UsbData->>UsbData: try_install_online_cache
  end
  UsbData-->>Daemon: install success/failure
  PortScan->>UsbData: resolve(vid, pid)
  UsbData-->>PortScan: vendor/product + cdc classification
Loading

Possibly related issues

Possibly related PRs

  • FastLED/fbuild#767: Both PRs modify the online USB VID:PID overlay plumbing, adding/using usb-vids.proto.zstd in usb/data.rs and port_scan.rs.
  • FastLED/fbuild#742: Both PRs are tightly connected via port_scan.rs's VID:PID resolver/overlay-fetching logic, extended here with CDC labeling.
  • FastLED/fbuild#801: This PR's acceptance-test compile-backend initialization ties directly to the CompileBackend global installation API introduced there.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/usb-vid-proto-online-data

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@zackees
zackees force-pushed the codex/usb-vid-proto-online-data branch 18 times, most recently from e86690f to cb0c99c Compare July 1, 2026 04:32
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Jul 1, 2026
@zackees
zackees force-pushed the codex/usb-vid-proto-online-data branch 9 times, most recently from 02acb32 to 5bc737f Compare July 1, 2026 06:05
@zackees
zackees force-pushed the codex/usb-vid-proto-online-data branch 19 times, most recently from ce74c01 to 7d334d8 Compare July 1, 2026 12:59
@zackees
zackees force-pushed the codex/usb-vid-proto-online-data branch from 7d334d8 to 40d9c9e Compare July 1, 2026 13:21
zackees and others added 3 commits July 1, 2026 06:53
Move the proto-first / JSON-fallback overlay download + install helper into
`fbuild_core::usb::populate_online_cache_from_paths(_and_urls)` so both
`fbuild port scan` and daemon-backed device paths share the same cold-cache
hydration path. The helper still runs the reqwest blocking client on a
dedicated OS thread (safe from Tokio) and uses the shared
`fbuild_core::http::blocking_client` bridge.

`fbuild-cli` now delegates to the shared helper; the local
fetch/cache-freshness scaffolding is deleted and the port_scan test that
exercised the local helper is replaced by a proto-404-then-JSON-200
integration test (kept in the CLI crate) plus a new
`populate_online_cache_falls_back_to_json_when_proto_is_missing` test
in `fbuild-core::usb::data`.

`fbuild-daemon` startup now spawn_blocking's a best-effort call to
`populate_online_cache_from_paths` using `<cache-root>/usb/{usb-vids.proto.zstd,
usb-vid.json}` before device refresh/list can be served, so
`fbuild device list/status/deploy` return full vendor + product names
instead of tier-1 vendor-only + `Device 0xPPPP` placeholder. Any failure
logs at debug/warn and never blocks daemon bootstrap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`ban_std_fs_in_async` fires on `std::fs::create_dir_all` inside
`crates/fbuild-daemon/src/**`, even though the whole helper is invoked
through `tokio::task::spawn_blocking`. The mkdir was defensive only —
`fbuild_core::usb::populate_online_cache_from_paths` already creates the
parent directory inside its fetch step before writing the cache file, so
the daemon-side mkdir is redundant.

Drop the `std::fs` call entirely rather than allowlist it or force an
async rewrite; the resolver still degrades to the embedded vendor archive
if both fetches fail and the cache dir stays empty.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zackees
zackees marked this pull request as ready for review July 1, 2026 16:30
@zackees
zackees merged commit b055b52 into main Jul 1, 2026
103 of 105 checks passed
@zackees
zackees deleted the codex/usb-vid-proto-online-data branch October 16, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

1 participant