Repository navigation
fix(writer): write Rust's file statistics into the footer (#664) - #706
Merged
Merged
Conversation
vortex-jni writes a FileStatistics segment, pointed to by the postscript: one ArrayStats per top-level field with Rust 0.90.0's PRUNING_STATS (min, max, sum, null count, NaN count). vortex-java wrote none, although the schema declares it. FileStatsAccumulator ports 0.90.0's FileStatsAccumulator: each stat is taken per writeChunk batch and folded at close, so - a UTF-8/binary bound is cut to 64 bytes per batch and is inexact when any batch cut it; - a UTF-8/binary max is dropped when any batch has none (all null, or no 64-byte upper bound); - a sum is dropped on overflow, a decimal(p) one past p+10 digits per batch and p+20 in the fold, serialized at the p+20 width; - extensions get storage min/max only, nested types the null count, variants nothing, and a column without rows an empty entry. Scalar kinds follow Rust (f16_value for F16, decimal LE bytes, u64 for unsigned and bool sums): Rust checks them against the dtype on open, so a wrong one makes the file unreadable. ZoneAccumulator's per-zone scan becomes the shared `extremes`, reused per batch; a non-nullable primitive batch uses the vectorized minMax. Cross-checked byte for byte against vortex-jni's footer for the same batches (FileStatisticsInteropIntegrationTest), and vortex-jni reads the Java file. Cost: one more pass over each batch on the caller thread, as Rust does. JavaVsJniWriteBenchmark.javaWrite (OHLC 10M rows, depth 3, interleaved, noisy machine): 1.08/1.18 -> 0.94/1.05 ops/s (~11%), mostly the sequential float sum (kept sequential for Rust's bits) and the string scan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Nov 8, 2026
dfa1
added a commit
that referenced
this pull request
Nov 8, 2026
…rtex-jni Every interop test compares what a file holds, never which optional FlatBuffer fields its metadata carries, so a field the Java writer leaves out fails nothing: the footer's file statistics went unwritten for releases (#706). This writes the same rows through both writers and asserts the set of fields present in the postscript, footer, dtype, layout and file statistics is equal, walking the generated readers by reflection (hasX() / non-null table accessors), so a field added to a schema is covered without editing the test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
added a commit
that referenced
this pull request
Nov 8, 2026
…rtex-jni Every interop test compares what a file holds, never which optional FlatBuffer fields its metadata carries, so a field the Java writer leaves out fails nothing: the footer's file statistics went unwritten for releases (#706). This writes the same rows through both writers and asserts the set of fields present in the postscript, footer, dtype, layout and file statistics is equal, walking the generated readers by reflection (hasX() / non-null table accessors), so a field added to a schema is covered without editing the test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of three PRs for #664 (write summary). Parity gap first: vortex-jni writes a
FileStatisticssegment (oneArrayStatsper top-level field: min, max, sum, null/NaN counts), vortex-java wrote none. The write summary (next PR) will report exactly these.FileStatsAccumulatorports Rust 0.90.0's per-batch fold (truncated string bounds, dropped max, overflow-checked sums, decimal widths). Behavior was probed empirically against vortex-jni 0.90.0, since Rustmainhas moved on (nested stats, no sum).FileStatisticsInteropIntegrationTest: byte-for-byte equal to vortex-jni's footer for 17 column kinds over two batches, plus a Null column; vortex-jni reads the Java file.ZoneAccumulator's scan is shared asextremes.Cost: ~11% slower
javaWrite(single-threaded default), the extra pass Rust also does; details in the commit message. Unrelated waste spotted:SegmentEncoderstill computes a per-segment sum (sumStat, ~1.3% of samples) undervortex.zoned, which writes no zone sum; a follow-up could drop it.🤖 Generated with Claude Code