Skip to content

Drop dependency on spatialbench-arrow - #11724

Merged
lwwmanning merged 4 commits into
developfrom
wm/spatialbench-modern-deps-399fe3
Nov 22, 2026
Merged

lwwmanning merged 4 commits into
developfrom
wm/spatialbench-modern-deps-399fe3

Conversation

@lwwmanning

@lwwmanning lwwmanning commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

This changes how vortex-bench builds the SpatialBench trip, building, and customer tables, and which SpatialBench crates it depends on. The zone table still comes from the external spatialbench-cli.

Before, SpatialBench used spatialbench and spatialbench-arrow 0.2.0, the latest release. spatialbench-arrow builds on Arrow 56, and the workspace's Parquet 59 writer can't take Arrow 56 batches. So wkb.rs wrote them through a second Parquet 56 copy, aliased as spatialbench-parquet. That kept 13 Arrow 56 and Parquet 56 crates in Cargo.lock, and Renovate's PRs to bump the alias could never build (#11320 is the latest).

After, the generators still come from spatialbench 0.2.0 on crates.io, which does not depend on Arrow. The Arrow conversion for the three tables is now in datagen/batches.rs, about 250 lines adapted from spatialbench-arrow 0.2.0 and built on the workspace Arrow. A SpatialBench release can no longer hold back an Arrow, Parquet, or DataFusion upgrade. Upstream took ten months to move from Arrow 56 to 58. wkb.rs now writes with the workspace parquet and the shared parquet_writer_properties() from #11720, and Cargo.lock has 20 fewer packages.

The cost is a copy that must keep upstream's schemas and values. Compare it with upstream when spatialbench changes version. Its header names the source tag. The remaining Parquet 58.4.1 copy comes from lance-bench, which needs it for lance 14.

At scale factors 0.1 and 1, the new code writes Parquet files that are byte-identical to those from spatialbench-arrow on the same Arrow and Parquet. Against the old Arrow 56 build, every table has the same schema, row count, values, and GeoParquet geo metadata. File sizes differ by less than 1%, which fits the writer changes between Parquet 56 and 59.

🤖 Generated with Claude Code

The latest spatialbench release (0.2.0) depends on arrow 56, so vortex-bench
wrote its batches through a second, parquet 56 copy. Point spatialbench and
spatialbench-arrow at the head of apache/sedona-spatialbench#146, which moves
them to arrow 59 without changing any generator code, and write the batches
with the workspace parquet. Cargo.lock drops every arrow 56 and parquet 56
crate.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning added the changelog/chore A trivial change label Nov 21, 2026
@lwwmanning
lwwmanning marked this pull request as ready for review November 21, 2026 22:34
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-22T00:01:59.439191Z 9ca1c69 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…wm/spatialbench-modern-deps-399fe3

Signed-off-by: Will Manning <will@willmanning.io>

# Conflicts:
#	vortex-bench/src/spatialbench/datagen/wkb.rs
The spatialbench generators do not depend on Arrow, but spatialbench-arrow
does, and its releases trail Arrow by months. Depending on it at any
version would hold back workspace Arrow, Parquet, and DataFusion upgrades.

Use spatialbench 0.2.0 from crates.io for the generators, and keep the
Arrow conversion for the building, customer, and trip tables in
vortex-bench, adapted from spatialbench-arrow 0.2.0. This drops the git
dependency and its cargo-deny exception. The generated Parquet files are
byte-identical to those from spatialbench-arrow on the same Arrow and
Parquet.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning changed the title Build SpatialBench data with the workspace's Arrow and Parquet Keep SpatialBench from holding back Arrow and Parquet upgrades Nov 22, 2026
@lwwmanning lwwmanning changed the title Keep SpatialBench from holding back Arrow and Parquet upgrades Drop dependency on spatialbench-arrow Nov 22, 2026
@lwwmanning
lwwmanning enabled auto-merge (squash) November 22, 2026 00:07
Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning
lwwmanning merged commit e44ca21 into develop Nov 22, 2026
77 checks passed
@lwwmanning
lwwmanning deleted the wm/spatialbench-modern-deps-399fe3 branch November 22, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants