Skip to content

fix(writer): no chunked layout around a single chunk, plus a metadata field-parity fitness test - #711

Merged
dfa1 merged 2 commits into
mainfrom
footer-field-parity
Nov 8, 2026
Merged

dfa1 merged 2 commits into
mainfrom
footer-field-parity

Conversation

@dfa1

@dfa1 dfa1 commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Follow-up to #706 ("how did we never spot it?"): a fitness function that would have.

  • test: FooterFieldParityIntegrationTest writes the same rows through vortex-jni and Java and asserts the set of present optional fields in the postscript, footer, dtype, layout and file statistics is equal. It walks the generated FlatBuffer readers by reflection, so new schema fields are covered automatically.
  • fix: on its first run it caught a real divergence: Java wrapped a single-chunk column (and single-chunk dict codes) in vortex.chunked; Rust 0.90.0 writes the chunk itself (ChunkedLayoutStrategy, child_layouts.len() == 1). Two global-dict tests asserted the old shape and are updated.

Separate commits per change. Not covered by the test: fields our .fbs copies lack entirely (schema drift vs the pinned Rust tag, e.g. the postscript's metadata segments) — that would need a schema comparison.

🤖 Generated with Claude Code

dfa1 and others added 2 commits November 8, 2026 14:05
Rust's ChunkedLayoutStrategy returns the only child itself when a
column has one chunk (vortex-layout chunked/writer.rs at 0.90.0:
`if child_layouts.len() == 1`). LayoutTree always wrapped the column's
flats, and a dict's codes, in vortex.chunked: one extra layout node per
single-chunk column, which every Rust and Java reader accepted, so no
interop test noticed. Found by FooterFieldParityIntegrationTest (next
commit).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rtex-jni

Every interop test compares what a file holds, never which optional
FlatBuffer fields its metadata carries, so a field the Java writer
leaves out fails nothing: the footer's file statistics went unwritten
for releases (#706). This writes the same rows through both writers and
asserts the set of fields present in the postscript, footer, dtype,
layout and file statistics is equal, walking the generated readers by
reflection (hasX() / non-null table accessors), so a field added to a
schema is covered without editing the test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 force-pushed the footer-field-parity branch from 5664cdb to 9becd2d Compare November 8, 2026 13:05
@dfa1
dfa1 merged commit 60fd800 into main Nov 8, 2026
6 checks passed
@dfa1
dfa1 deleted the footer-field-parity branch November 8, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant