Skip to content

fix(writer): every primitive column can get Rust's dict layout, narrow widths included (#651) - #703

Merged
dfa1 merged 1 commit into
mainfrom
fix/651-narrow-int-dict
Nov 8, 2026
Merged

dfa1 merged 1 commit into
mainfrom
fix/651-narrow-int-dict

Conversation

@dfa1

@dfa1 dfa1 commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Part of #651.

Rust's dict layout admits every primitive (`dict_layout_supported`) and leaves the decision to its probe. Java admitted only I32/U32, I64/U64 and F64, so 8-bit, 16-bit and F32 columns never reached the probe.

  • Public BI Generico `Disco` (nullable SMALLINT, first 5M rows): Rust 3,913,600 B, Java before 5,633,456 B (1.44x), Java after 3,912,232 B.
  • Found with the compressor JFR events: the recording had no global-dict probe (`segment = -1`) for `Disco`. After the fix the probe picks `vortex.dict`.
  • `DictLayoutInteropIntegrationTest#admitsANullableNarrowColumnAsRustDoes` (I8/I16/U16, shaped like `Disco`) compares the encoding tree with vortex-jni node for node and has vortex-jni read Java's file. It fails without the fix.

🤖 Generated with Claude Code

…w widths included (#651)

Rust's dict layout admits every primitive (dict_layout_supported) and
lets its probe decide. Java filtered by ptype first (I32/U32, I64/U64,
F64 only), so 8- and 16-bit and F32 columns never reached the probe.
Public BI Generico's `Disco` (a nullable SMALLINT, 40% valid, 62
distinct values) came out 1.44x larger: bit-packed with a validity per
chunk, where Rust writes one values pool and run-end codes.

Found with the compressor's JFR events: the Winner/Candidate recording
of the import had no global-dict probe (segment -1) for Disco, while
the I32 columns next to it had one. After the fix the probe picks
vortex.dict (2.54 estimated, 2.39 achieved) and the column is 3.91 MB,
as Rust's.

DictColumnState now buffers every carrier (byte[], short[] for
I16/U16/F16, float[]); the dead valueBits helper is removed. The
exclusion's original reasons no longer hold: the reader decodes a
narrow-int dict (it reads Rust's Disco file), and Rust does dict them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 merged commit 75f8ee3 into main Nov 8, 2026
6 checks passed
@dfa1
dfa1 deleted the fix/651-narrow-int-dict branch November 8, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant