You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
VortexWriter writes zone maps as the vortex.stats layout (VortexWriter.java, LayoutId.STATS). Rust's default writer (0.86.1) emits vortex.zoned (core2026.08.0), with aggregate-spec metadata: max/min (skip NaNs) for most columns, bounded_max/bounded_min for Utf8/Binary, plus null_count and nan_count (vortex-layout/src/layouts/zoned/writer.rs, default_zoned_aggregate_fns).
Rust still reads vortex.stats (it is in core2025.05.0), so nothing breaks today. But this diverges from the prime directive (write what Rust writes), and docs/compatibility.md does not record it.
Fix: emit vortex.zoned with Rust's default aggregate set when the targeted edition includes core2026.08.0, and keep vortex.stats for older edition targets. Depends on supporting the bounded aggregates (see the related zone-map ticket).
Layout metadata is byte-identical to vortex-jni's for F64 and F32.
Older edition targets (before core2026.08.0) keep the legacy vortex.stats.
Open decision: Rust's zoned layout has no per-zone sum, and its zones don't line up with chunks. So Calcite's zone fold and ZoneReducer (which need one zone per chunk, plus sums for SUM) would fall back to plain scans on new Java-written files. Results stay correct, push-down is lost. Rust-written files already have this limitation. About 45 writer tests assume the legacy one-zone-per-chunk layout and need to move to an older edition or to data of at least 8192 rows.
Options:
A: ship zoned now, then generalize the fold to work on zones.
Activity
dfa1 commentedon Oct 3, 2026
Depends on #446 (bounded aggregate support).
dfa1 commentedon Oct 8, 2026
Work in progress on branch
wip/447-zoned-layout(9f956a7).Done there:
vortex.zonedwith fixed 8192-row zones, independent ofwriteChunkbatches. A batch's last partial zone carries over into the next batch.bounded_max/bounded_min(64 bytes, Rust's truncation rules),null_countmax/min(skip NaN),nan_count,null_countmax/min,null_countnull_countonlycore2026.08.0) keep the legacyvortex.stats.Open decision: Rust's zoned layout has no per-zone
sum, and its zones don't line up with chunks. So Calcite's zone fold andZoneReducer(which need one zone per chunk, plus sums forSUM) would fall back to plain scans on new Java-written files. Results stay correct, push-down is lost. Rust-written files already have this limitation. About 45 writer tests assume the legacy one-zone-per-chunk layout and need to move to an older edition or to data of at least 8192 rows.Options:
vortex.statsas the default.wip(writer): cascade F32 ALP ints and dict values, as Rust (#458)
docs(changelog): Breaking marks API breaks only; written files stay r…
docs(changelog): Breaking marks API breaks only; written files stay r…
feat(calcite): fold filtered aggregates over Rust's vortex.zoned zones
fix(performance): Calcite boundary benchmark times the fold again
feat(calcite): fold filtered aggregates over Rust's vortex.zoned zones
fix(performance): Calcite boundary benchmark times the fold again
perf(writer): sum a segment only for the legacy zone map that reads it
perf(writer): sum a segment only for the legacy zone map that reads it