Skip to content

fix(ci): prefetch LibTorch for every release platform and package it reliably - #5

Open
fffonion wants to merge 1 commit into
masterfrom
subagent/flint-release-env-20261117
Open

fffonion wants to merge 1 commit into
masterfrom
subagent/flint-release-env-20261117

Conversation

@fffonion

Copy link
Copy Markdown
Contributor

Why

Release run 43928665884 (tag 0.2.0) failed on all three platforms. Two separate root causes, both LibTorch related.

Linux — packaging could not find LibTorch (build succeeded)

RuntimeError: expected one tch-rs LibTorch directory, found [];
checked torch-sys build output under /home/runner/work/flint/flint/flint/target/release/build

5fe0b71 dropped the prefetch step and relied on tch's own download. Two problems with that:

  • The download lands in the torch-sys OUT_DIR (.../build/torch-sys/<hash>/out/libtorch), which the packaging step never looked at.
  • Current cargo no longer writes build/torch-sys-<hash>/output; it nests build dirs per crate as build/torch-sys/<hash>/output. The same run shows the macOS job using that layout (target/release/build/torch-sys/08dcea37e4771c48/out/build_script_build), so the torch-sys-*/output glob matched zero files — hence found [] instead of a per-platform message.

macOS arm64 — could not build at all

Failed to retrieve torch from pypi. Pre-built version of libtorch for apple silicon are not available.

tch 0.24 → torch-sys 0.24 publishes no Apple Silicon LibTorch; its fallback path asks PyPI for a CPython wheel and there is none. Only LIBTORCH pointing at an extracted libtorch-macos-arm64-2.11.0.zip avoids that branch.

Windows

Still running in the failing run, but the same mechanism applies: the build depended on tch downloading LibTorch itself into its OUT_DIR, so the packaging step would not have found the DLLs either.

What changed (.github/workflows/release.yml only)

Prefetch LibTorch on all three platforms. Downloads into target/store/libtorch/<version>/<device>/libtorch:

platform device archive
linux-x86_64 cpu libtorch-shared-with-deps-2.11.0+cpu.zip
macos-arm64 cpu libtorch-macos-arm64-2.11.0.zip (MPS included)
windows-x86_64 cu130 libtorch-win-shared-with-deps-2.11.0+cu130.zip

LIBTORCH_VERSION is pinned to 2.11.0, which is TORCH_VERSION in torch-sys 0.24 — torch-sys compares it against libtorch/build-version and aborts on a mismatch, so the step verifies the archive's build-version itself and fails with a clear message instead of a build-time surprise. The step is idempotent (re-uses an already extracted tree) and retries transient CDN failures.

Exported for the following steps: LIBTORCH, LD_LIBRARY_PATH (Linux) / DYLD_LIBRARY_PATH (macOS) / PATH (Windows). torch-sys takes LIBTORCH first and skips every download path, so build time and packaging use the same libraries.

Packaging (Package runner) resolves LibTorch properly.

  1. $LIBTORCH/lib first — authoritative, same tree the build used.
  2. Otherwise parse the torch-sys build script output for cargo:rustc-link-search=native=, cargo:rustc-link-search=, cargo:libtorch_lib=, scanning both build/torch-sys-*/output (legacy) and build/torch-sys/*/output plus a recursive glob.
  3. If no candidate contains the platform's main library, fail with one explicit message naming LIBTORCH, the build outputs scanned, whether the build dir exists, and every candidate checked.

It then copies all LibTorch shared libraries (.so* / .dylib / .dll), keeps the patchelf --set-rpath $ORIGIN and install_name_tool -add_rpath @executable_path handling (macOS now skips the rpath insert when it is already present), and adds guards: the cu130 build must bundle a cudart64_*.dll, the loader must resolve the bundled LibTorch libraries (Linux ldd, macOS otool -L rejecting leaked /Users/ build paths), and the archive is rejected if it reaches GitHub's 2 GiB release-asset limit.

macOS runner label is unchanged (macos-latest), and koharu-torch is not restored.

Verification done here

Local only — no three-platform build was run, so this does not claim native CI success.

  • actionlint clean; yaml.safe_load parses; both shell: python blocks compile.
  • Prefetch step executed against the simulated platform environments for Linux/macOS/Windows, using the real published archive file lists as the payload: 7/7 checks pass (URL per platform, required libraries present in the real archives, build-version preserved, LIBTORCH/LD_LIBRARY_PATH/DYLD_LIBRARY_PATH/GITHUB_PATH written, idempotent re-run with zero downloads, mismatched build-version rejected).
  • Real Linux prefetch against download.pytorch.org: 814 MB extracted, build-version = 2.11.0+cpu, all required libraries present.
  • Real torch-sys 0.24.0 + tch 0.24.0 build with LIBTORCH pointing at that tree: compiles and links with no PyPI or download.pytorch.org access in the log, and the emitted build output starts with cargo:rustc-link-search=native=<prefetched>/lib — the exact prefix the packaging step parses.
  • Packaging simulation, 8/8: real Linux run (real patchelf, real torch-linked binary) produces an archive whose binary runs from a clean environment with $ORIGIN rpath resolving libtorch_cpu.so, libc10.so and libgomp-98b21ff3.so.1 out of the package; fallback resolution for both cargo build-dir layouts; explicit failure when nothing is discoverable; macOS and Windows paths (dylib/DLL bundling, @executable_path rpath, CUDA DLLs next to flint.exe); CUDA bundle without cudart64_*.dll rejected; every required library name confirmed to exist in the published archives.

Known risk for the next run

The Windows cu130 package carries ~2.66 GB of DLLs uncompressed (≈1.87 GB zipped) because PyTorch's CUDA LibTorch bundles the whole CUDA runtime (cuBLAS, cuDNN, cuFFT, cuSOLVER, NVRTC…). Together with flint.exe and the koharu libraries it is close to GitHub's 2 GiB per-asset limit — the step now fails with an explicit size message rather than an opaque upload error. If the real binary pushes it over, the follow-up is to split the Windows asset (CUDA runtime as a second archive) rather than to drop CUDA.

…reliably

The 0.2.0 release run (43928665884) failed on all three platforms for two
distinct reasons:

* Linux built fine but packaging could not find LibTorch: current cargo
  stores build script output under target/release/build/torch-sys/<hash>/
  (nested per crate), so the torch-sys-*/output glob matched nothing and the
  step failed with "found []". LibTorch had only ever been downloaded inside
  the torch-sys OUT_DIR, where packaging cannot reach it.
* macOS arm64 could not build at all: tch 0.24 -> torch-sys 0.24 has no
  pre-built LibTorch for Apple Silicon, so its fallback tries to fetch a
  CPython wheel from PyPI and aborts with "Failed to retrieve torch from
  pypi. Pre-built version of libtorch for apple silicon are not available."
  Windows would have hit the same class of failure.

Prefetch LibTorch once per platform into target/store/libtorch/<version>/
<device>/libtorch (cpu on Linux/macOS, cu130 on Windows), verify
libtorch/build-version against the pinned version, and export LIBTORCH plus
LD_LIBRARY_PATH / DYLD_LIBRARY_PATH / PATH so torch-sys never downloads
anything itself and build time and packaging see the same libraries.

Packaging now resolves the library directory from LIBTORCH first, falls back
to the torch-sys build output (both cargo build-dir layouts), and fails with
an explicit message when nothing is found. It bundles the LibTorch shared
libraries, keeps the patchelf/install_name_tool rpath rewrite, verifies the
CUDA runtime is bundled for the cu130 build, checks the loader can resolve
the bundled libraries, and rejects archives over the 2 GiB release asset
limit.

koharu-torch is not restored; packaging still ships the direct tch/koharu
libraries only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant