You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Writer: integer cascade choices differ from Rust's on real columns (run-end, nullable bit-packing), 1.2–1.8x larger #651
vortex-java's Parquet import vs the vortex-jni-written file, data bytes; the trees are each side's two most common chunk encodings:
dataset
column
type
Rust
Java
Rust tree
Java tree
clickbench-hits
RefererRegionID
I32
68.1 MB
124.7 MB (1.83x)
171x bitpacked(patches), 12x runend
274x bitpacked(patches), 39x runend
clickbench-hits
FUniqID
I64
255.4 MB
358.3 MB (1.40x)
289x bitpacked, 176x primitive
365x runend(for(bitpacked), primitive)
clickbench-hits
UserID
I64
274.5 MB
366.4 MB (1.33x)
433x primitive, 116x runend(bitpacked, primitive)
367x runend(for(bitpacked), primitive)
bi-generico
Hora_Pagina
I32?
207.3 MB
269.1 MB (1.30x)
158x primitive, 109x bitpacked
371x bitpacked(bool)
bi-generico
Disco
I16?
92.8 MB
119.9 MB (1.29x)
159x runend(for(bitpacked), bitpacked)
183x bitpacked(bool)
bi-generico
Codigo
I32?
249.8 MB
289.4 MB (1.16x)
272x primitive
263x bitpacked(patches)
Java picks run-end where Rust keeps bit-packed/primitive (and vice versa), and its nullable chunks pay a vortex.bool validity on every bit-packed chunk. Likely several sampling/estimate differences in the cascade; each column above is a reproducer to bisect (the encoding-tree interop tests take a column shape and compare trees). Some of RefererRegionID may be #649 or chunk count (Java writes more chunks of these columns).
Re-measured on current main, with the compressor JFR events
Setup: each column was sliced from the source into its own Parquet file and written twice: by Rust (vortex-data0.90.0, the same Rust release as our vortex-jni) and by vortex import, with the JFR events on:
Version note: the corpus .vortex files in this ticket's table were built by Raincloud's venv, which has vortex_data0.86.1. For these columns 0.86.1 writes within a few bytes of 0.90, so the version difference doesn't explain the gap. The gap came from Java choices since fixed (#649 numeric dict layout, #470 repartition) and from the bug below.
Winner events with segment = -1 are the global-dict admission probe. The Disco recording had none, while the other integer columns had one each:
probe (segment -1) Winner events: Disco 0 Hora_Pagina 3 Codigo 3
Disco root winners: 12x fastlanes.bitpacked, 1x vortex.dict, 1x vortex.runend
So Disco was ruled out before the probe ran. That pointed straight at the pre-probe filter, DictColumnState.admitsPtype, which admitted only I32/U32, I64/U64 and F64, while Rust's dict_layout_supported admits every primitive. The diffed trees agreed: Rust wrote one primitive(bool) values pool plus runend(for(bitpacked), bitpacked) codes, and Java wrote bitpacked(bool) per chunk. That is exactly the "Rust tree" / "Java tree" pair in this ticket's table.
After #703 the probe appears, and vortex.dict wins on the stats estimate and on the measured encode:
Column Segment Cascade Path Depth Parent Scheme Input Compressed Estimated Achieved Accepted
Disco -1 0 vortex.dict 17.0 kB 7.1 kB 2.538 2.387 true
Disco -1 dict.0 1 dict fastlanes.bitpacked 9.0 kB 7.0 kB 1.286 1.286 true
Disco -1 dict.1 1 dict fastlanes.for 124 B 896 B 2.286 0.138 false (not smaller than the uncompressed array)
Disco root winners after: 10x vortex.runend, 2x vortex.dict, 2x fastlanes.bitpacked
The last row also shows what accepted = false is for: FoR on a 124-byte code array was estimated at 2.29x but measured at 0.14x (bit-packing padding), and the compressor kept the canonical array.
What's left, and small: the zone-map statistics table of Hora_Pagina, Codigo and Disco is encoded slightly differently (Java bit-packs those min/max columns with patches, where Rust keeps vortex.dict over primitive). It amounts to bytes per file and is #583's territory.
Fixed in 2fcf17c. The remaining zone-table gap was the dictionary's value order: Rust's IntDictScheme/FloatDictScheme take the values in the iteration order of their hashbrown + FxHash distinct-values map, Java took them first-seen. Same values under other codes gave a different code bit-width histogram, so the codes bit-packed with patches (Disco) or bit-packed where Rust kept them primitive (Hora_Pagina). It shows only at 5M rows; at 2.4M the two orders happened to agree.
DictValueOrder now replays that map (FxHasher, hashbrown 0.17 probing, growth and iteration; group width from os.arch, as Rust's SSE2/NEON). On the first 5M rows of Generico, Codigo, Disco and Hora_Pagina now write vortex-jni's encoding trees, data and zone table, in one batch or 64K batches. DictValueOrderInteropIntegrationTest compares every buffer byte for byte for the dense, hashed and float collectors, and fails without the fix.
5e19246: an unsigned 8/16/32-bit dictionary whose values straddle the type's top bit took Rust's hashed counter instead of the dense one (Java's sign-extended bits). New byte-for-byte vortex-jni cross-checks for the corners that had none: that case, a float set filled exactly and grown by a duplicate, and integer keys piling up past 16 buckets (where hashbrown probes by group width, so each CI host checks its own).
45eb71e + f08a388: the dict layout's codes width for 8-bit columns. Rust 0.91 (vortex-jni moved to 0.91.0 in a368b9d) gives U8 codes only to non-nullable 8-bit values and U16 to nullable ones and wider types; Java always wrote U16. 45eb71e applied 0.90's rule and was corrected by f08a388; DictLayoutInteropIntegrationTest now compares every narrow type, nullable and not, byte for byte.
The write benchmark shows no regression from the dictionary ordering (javaWrite depth 3, 3 forks, interleaved: 0.825/0.768 ops/s before, 0.891/0.956 after).
Activity
test(integration): say what the #651-style shapes are
test(writer): pin the integer cascade's Rust-parity choices (#651)
test(writer): pin the integer cascade's Rust-parity choices (#651)
dfa1 commentedon Nov 8, 2026
Re-measured on current
main, with the compressor JFR eventsSetup: each column was sliced from the source into its own Parquet file and written twice: by Rust (
vortex-data0.90.0, the same Rust release as our vortex-jni) and byvortex import, with the JFR events on:Version note: the corpus
.vortexfiles in this ticket's table were built by Raincloud's venv, which hasvortex_data0.86.1. For these columns 0.86.1 writes within a few bytes of 0.90, so the version difference doesn't explain the gap. The gap came from Java choices since fixed (#649 numeric dict layout, #470 repartition) and from the bug below.RefererRegionIDFUniqIDUserIDHora_PaginaCodigoDiscoHow JFR found the
DiscobugWinnerevents withsegment = -1are the global-dict admission probe. TheDiscorecording had none, while the other integer columns had one each:So
Discowas ruled out before the probe ran. That pointed straight at the pre-probe filter,DictColumnState.admitsPtype, which admitted only I32/U32, I64/U64 and F64, while Rust'sdict_layout_supportedadmits every primitive. The diffed trees agreed: Rust wrote oneprimitive(bool)values pool plusrunend(for(bitpacked), bitpacked)codes, and Java wrotebitpacked(bool)per chunk. That is exactly the "Rust tree" / "Java tree" pair in this ticket's table.After #703 the probe appears, and
vortex.dictwins on the stats estimate and on the measured encode:The last row also shows what
accepted = falseis for: FoR on a 124-byte code array was estimated at 2.29x but measured at 0.14x (bit-packing padding), and the compressor kept the canonical array.What's left, and small: the zone-map statistics table of
Hora_Pagina,CodigoandDiscois encoded slightly differently (Java bit-packs those min/max columns with patches, where Rust keepsvortex.dictoverprimitive). It amounts to bytes per file and is #583's territory.fix(writer): every primitive column can get Rust's dict layout, narro…
fix(writer): order a cascade dictionary's values as Rust does (#651)
dfa1 commentedon Nov 8, 2026
Fixed in 2fcf17c. The remaining zone-table gap was the dictionary's value order: Rust's
IntDictScheme/FloatDictSchemetake the values in the iteration order of their hashbrown + FxHash distinct-values map, Java took them first-seen. Same values under other codes gave a different code bit-width histogram, so the codes bit-packed with patches (Disco) or bit-packed where Rust kept them primitive (Hora_Pagina). It shows only at 5M rows; at 2.4M the two orders happened to agree.DictValueOrdernow replays that map (FxHasher, hashbrown 0.17 probing, growth and iteration; group width fromos.arch, as Rust's SSE2/NEON). On the first 5M rows of Generico,Codigo,DiscoandHora_Paginanow write vortex-jni's encoding trees, data and zone table, in one batch or 64K batches.DictValueOrderInteropIntegrationTestcompares every buffer byte for byte for the dense, hashed and float collectors, and fails without the fix.fix(writer): count a narrow unsigned dictionary densely across its si…
fix(writer): a dict layout over 8-bit values has U8 codes, as Rust's (#…
fix(writer): a nullable 8-bit dict layout keeps U16 codes, as Rust 0.…
dfa1 commentedon Nov 8, 2026
Follow-ups after review, all on main:
DictLayoutInteropIntegrationTestnow compares every narrow type, nullable and not, byte for byte.The write benchmark shows no regression from the dictionary ordering (javaWrite depth 3, 3 forks, interleaved: 0.825/0.768 ops/s before, 0.891/0.956 after).
test(integration): Java reads back its own narrow dict layout (#651)