Repository navigation
Drop dependency on spatialbench-arrow - #11724
Merged
Merged
Conversation
The latest spatialbench release (0.2.0) depends on arrow 56, so vortex-bench wrote its batches through a second, parquet 56 copy. Point spatialbench and spatialbench-arrow at the head of apache/sedona-spatialbench#146, which moves them to arrow 59 without changing any generator code, and write the batches with the workspace parquet. Cargo.lock drops every arrow 56 and parquet 56 crate. Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning
marked this pull request as ready for review
November 21, 2026 22:34
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…wm/spatialbench-modern-deps-399fe3 Signed-off-by: Will Manning <will@willmanning.io> # Conflicts: # vortex-bench/src/spatialbench/datagen/wkb.rs
The spatialbench generators do not depend on Arrow, but spatialbench-arrow does, and its releases trail Arrow by months. Depending on it at any version would hold back workspace Arrow, Parquet, and DataFusion upgrades. Use spatialbench 0.2.0 from crates.io for the generators, and keep the Arrow conversion for the building, customer, and trip tables in vortex-bench, adapted from spatialbench-arrow 0.2.0. This drops the git dependency and its cargo-deny exception. The generated Parquet files are byte-identical to those from spatialbench-arrow on the same Arrow and Parquet. Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning
enabled auto-merge (squash)
November 22, 2026 00:07
Signed-off-by: Will Manning <will@willmanning.io>
robert3005
approved these changes
Nov 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changes how
vortex-benchbuilds the SpatialBenchtrip,building, andcustomertables, and which SpatialBench crates it depends on. Thezonetable still comes from the externalspatialbench-cli.Before, SpatialBench used
spatialbenchandspatialbench-arrow0.2.0, the latest release.spatialbench-arrowbuilds on Arrow 56, and the workspace's Parquet 59 writer can't take Arrow 56 batches. Sowkb.rswrote them through a second Parquet 56 copy, aliased asspatialbench-parquet. That kept 13 Arrow 56 and Parquet 56 crates inCargo.lock, and Renovate's PRs to bump the alias could never build (#11320 is the latest).After, the generators still come from
spatialbench0.2.0 on crates.io, which does not depend on Arrow. The Arrow conversion for the three tables is now indatagen/batches.rs, about 250 lines adapted fromspatialbench-arrow0.2.0 and built on the workspace Arrow. A SpatialBench release can no longer hold back an Arrow, Parquet, or DataFusion upgrade. Upstream took ten months to move from Arrow 56 to 58.wkb.rsnow writes with the workspaceparquetand the sharedparquet_writer_properties()from #11720, andCargo.lockhas 20 fewer packages.The cost is a copy that must keep upstream's schemas and values. Compare it with upstream when
spatialbenchchanges version. Its header names the source tag. The remaining Parquet 58.4.1 copy comes fromlance-bench, which needs it for lance 14.At scale factors 0.1 and 1, the new code writes Parquet files that are byte-identical to those from
spatialbench-arrowon the same Arrow and Parquet. Against the old Arrow 56 build, every table has the same schema, row count, values, and GeoParquetgeometadata. File sizes differ by less than 1%, which fits the writer changes between Parquet 56 and 59.🤖 Generated with Claude Code