Repository navigation
fix(writer): every primitive column can get Rust's dict layout, narrow widths included (#651) - #703
Merged
Merged
Conversation
…w widths included (#651) Rust's dict layout admits every primitive (dict_layout_supported) and lets its probe decide. Java filtered by ptype first (I32/U32, I64/U64, F64 only), so 8- and 16-bit and F32 columns never reached the probe. Public BI Generico's `Disco` (a nullable SMALLINT, 40% valid, 62 distinct values) came out 1.44x larger: bit-packed with a validity per chunk, where Rust writes one values pool and run-end codes. Found with the compressor's JFR events: the Winner/Candidate recording of the import had no global-dict probe (segment -1) for Disco, while the I32 columns next to it had one. After the fix the probe picks vortex.dict (2.54 estimated, 2.39 achieved) and the column is 3.91 MB, as Rust's. DictColumnState now buffers every carrier (byte[], short[] for I16/U16/F16, float[]); the dead valueBits helper is removed. The exclusion's original reasons no longer hold: the reader decodes a narrow-int dict (it reads Rust's Disco file), and Rust does dict them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #651.
Rust's dict layout admits every primitive (`dict_layout_supported`) and leaves the decision to its probe. Java admitted only I32/U32, I64/U64 and F64, so 8-bit, 16-bit and F32 columns never reached the probe.
🤖 Generated with Claude Code