Skip to content

Writer emits legacy vortex.stats layout; Rust's default writer emits vortex.zoned #447

Closed

Description

@dfa1

VortexWriter writes zone maps as the vortex.stats layout (VortexWriter.java, LayoutId.STATS). Rust's default writer (0.86.1) emits vortex.zoned (core2026.08.0), with aggregate-spec metadata: max/min (skip NaNs) for most columns, bounded_max/bounded_min for Utf8/Binary, plus null_count and nan_count (vortex-layout/src/layouts/zoned/writer.rs, default_zoned_aggregate_fns).

Rust still reads vortex.stats (it is in core2025.05.0), so nothing breaks today. But this diverges from the prime directive (write what Rust writes), and docs/compatibility.md does not record it.

Fix: emit vortex.zoned with Rust's default aggregate set when the targeted edition includes core2026.08.0, and keep vortex.stats for older edition targets. Depends on supporting the bounded aggregates (see the related zone-map ticket).

🤖 Generated with Claude Code

Activity

  1. added
    bugSomething isn't working
    on Oct 3, 2026
  2. dfa1 commented on Oct 3, 2026

    @dfa1
    OwnerAuthor

    Depends on #446 (bounded aggregate support).

  3. dfa1 commented on Oct 8, 2026

    @dfa1
    OwnerAuthor

    Work in progress on branch wip/447-zoned-layout (9f956a7).

    Done there:

    • Writes vortex.zoned with fixed 8192-row zones, independent of writeChunk batches. A batch's last partial zone carries over into the next batch.
    • Per-dtype aggregate set, taken from vortex-jni 0.86.1 output:
      • Utf8/Binary: bounded_max/bounded_min (64 bytes, Rust's truncation rules), null_count
      • floats: max/min (skip NaN), nan_count, null_count
      • bool/int/date/timestamp/decimal: max/min, null_count
      • list: null_count only
    • Layout metadata is byte-identical to vortex-jni's for F64 and F32.
    • Older edition targets (before core2026.08.0) keep the legacy vortex.stats.

    Open decision: Rust's zoned layout has no per-zone sum, and its zones don't line up with chunks. So Calcite's zone fold and ZoneReducer (which need one zone per chunk, plus sums for SUM) would fall back to plain scans on new Java-written files. Results stay correct, push-down is lost. Rust-written files already have this limitation. About 45 writer tests assume the legacy one-zone-per-chunk layout and need to move to an older edition or to data of at least 8192 rows.

    Options:

    • A: ship zoned now, then generalize the fold to work on zones.
    • B: generalize the fold first, then switch.
    • C: keep vortex.stats as the default.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions