Skip to content

fix(writer): write Rust's file statistics into the footer (#664) - #706

Merged
dfa1 merged 1 commit into
mainfrom
file-statistics
Nov 8, 2026
Merged

dfa1 merged 1 commit into
mainfrom
file-statistics

Conversation

@dfa1

@dfa1 dfa1 commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Second of three PRs for #664 (write summary). Parity gap first: vortex-jni writes a FileStatistics segment (one ArrayStats per top-level field: min, max, sum, null/NaN counts), vortex-java wrote none. The write summary (next PR) will report exactly these.

  • FileStatsAccumulator ports Rust 0.90.0's per-batch fold (truncated string bounds, dropped max, overflow-checked sums, decimal widths). Behavior was probed empirically against vortex-jni 0.90.0, since Rust main has moved on (nested stats, no sum).
  • FileStatisticsInteropIntegrationTest: byte-for-byte equal to vortex-jni's footer for 17 column kinds over two batches, plus a Null column; vortex-jni reads the Java file.
  • ZoneAccumulator's scan is shared as extremes.

Cost: ~11% slower javaWrite (single-threaded default), the extra pass Rust also does; details in the commit message. Unrelated waste spotted: SegmentEncoder still computes a per-segment sum (sumStat, ~1.3% of samples) under vortex.zoned, which writes no zone sum; a follow-up could drop it.

🤖 Generated with Claude Code

vortex-jni writes a FileStatistics segment, pointed to by the
postscript: one ArrayStats per top-level field with Rust 0.90.0's
PRUNING_STATS (min, max, sum, null count, NaN count). vortex-java wrote
none, although the schema declares it.

FileStatsAccumulator ports 0.90.0's FileStatsAccumulator: each stat is
taken per writeChunk batch and folded at close, so
- a UTF-8/binary bound is cut to 64 bytes per batch and is inexact when
  any batch cut it;
- a UTF-8/binary max is dropped when any batch has none (all null, or
  no 64-byte upper bound);
- a sum is dropped on overflow, a decimal(p) one past p+10 digits per
  batch and p+20 in the fold, serialized at the p+20 width;
- extensions get storage min/max only, nested types the null count,
  variants nothing, and a column without rows an empty entry.

Scalar kinds follow Rust (f16_value for F16, decimal LE bytes, u64 for
unsigned and bool sums): Rust checks them against the dtype on open, so
a wrong one makes the file unreadable.

ZoneAccumulator's per-zone scan becomes the shared `extremes`, reused
per batch; a non-nullable primitive batch uses the vectorized minMax.

Cross-checked byte for byte against vortex-jni's footer for the same
batches (FileStatisticsInteropIntegrationTest), and vortex-jni reads
the Java file.

Cost: one more pass over each batch on the caller thread, as Rust
does. JavaVsJniWriteBenchmark.javaWrite (OHLC 10M rows, depth 3,
interleaved, noisy machine): 1.08/1.18 -> 0.94/1.05 ops/s (~11%), mostly
the sequential float sum (kept sequential for Rust's bits) and the
string scan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 merged commit 1ba4ead into main Nov 8, 2026
6 checks passed
@dfa1
dfa1 deleted the file-statistics branch November 8, 2026 12:42
dfa1 added a commit that referenced this pull request Nov 8, 2026
…rtex-jni

Every interop test compares what a file holds, never which optional
FlatBuffer fields its metadata carries, so a field the Java writer
leaves out fails nothing: the footer's file statistics went unwritten
for releases (#706). This writes the same rows through both writers and
asserts the set of fields present in the postscript, footer, dtype,
layout and file statistics is equal, walking the generated readers by
reflection (hasX() / non-null table accessors), so a field added to a
schema is covered without editing the test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1 added a commit that referenced this pull request Nov 8, 2026
…rtex-jni

Every interop test compares what a file holds, never which optional
FlatBuffer fields its metadata carries, so a field the Java writer
leaves out fails nothing: the footer's file statistics went unwritten
for releases (#706). This writes the same rows through both writers and
asserts the set of fields present in the postscript, footer, dtype,
layout and file statistics is equal, walking the generated readers by
reflection (hasX() / non-null table accessors), so a field added to a
schema is covered without editing the test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant