Skip to content

Writer: integer cascade choices differ from Rust's on real columns (run-end, nullable bit-packing), 1.2–1.8x larger #651

Closed

Description

@dfa1

vortex-java's Parquet import vs the vortex-jni-written file, data bytes; the trees are each side's two most common chunk encodings:

dataset column type Rust Java Rust tree Java tree
clickbench-hits RefererRegionID I32 68.1 MB 124.7 MB (1.83x) 171x bitpacked(patches), 12x runend 274x bitpacked(patches), 39x runend
clickbench-hits FUniqID I64 255.4 MB 358.3 MB (1.40x) 289x bitpacked, 176x primitive 365x runend(for(bitpacked), primitive)
clickbench-hits UserID I64 274.5 MB 366.4 MB (1.33x) 433x primitive, 116x runend(bitpacked, primitive) 367x runend(for(bitpacked), primitive)
bi-generico Hora_Pagina I32? 207.3 MB 269.1 MB (1.30x) 158x primitive, 109x bitpacked 371x bitpacked(bool)
bi-generico Disco I16? 92.8 MB 119.9 MB (1.29x) 159x runend(for(bitpacked), bitpacked) 183x bitpacked(bool)
bi-generico Codigo I32? 249.8 MB 289.4 MB (1.16x) 272x primitive 263x bitpacked(patches)

Java picks run-end where Rust keeps bit-packed/primitive (and vice versa), and its nullable chunks pay a vortex.bool validity on every bit-packed chunk. Likely several sampling/estimate differences in the cascade; each column above is a reproducer to bisect (the encoding-tree interop tests take a column shape and compare trees). Some of RefererRegionID may be #649 or chunk count (Java writes more chunks of these columns).

Activity

  1. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Re-measured on current main, with the compressor JFR events

    Setup: each column was sliced from the source into its own Parquet file and written twice: by Rust (vortex-data 0.90.0, the same Rust release as our vortex-jni) and by vortex import, with the JFR events on:

    java -XX:StartFlightRecording:filename=rec.jfr,+io.github.dfa1.vortex.Candidate#enabled=true,+io.github.dfa1.vortex.Winner#enabled=true,+io.github.dfa1.vortex.Compress#enabled=true \
         -jar vortex-cli-all.jar import col.parquet col.vortex
    

    Version note: the corpus .vortex files in this ticket's table were built by Raincloud's venv, which has vortex_data 0.86.1. For these columns 0.86.1 writes within a few bytes of 0.90, so the version difference doesn't explain the gap. The gap came from Java choices since fixed (#649 numeric dict layout, #470 repartition) and from the bug below.

    column rows Rust 0.90 Java (main) encoding trees
    clickbench RefererRegionID 10M 9,444,240 9,440,860 (0.999x) identical
    clickbench FUniqID 10M 19,237,344 19,236,324 (0.999x) identical
    clickbench UserID 10M 19,598,320 19,597,402 (0.999x) identical
    Generico Hora_Pagina 5M 6,888,548 6,887,405 (0.999x) data identical, zone table differs
    Generico Codigo 5M 10,691,848 10,690,746 (0.999x) data identical, zone table differs
    Generico Disco 5M 3,913,600 5,633,456 (1.44x) → 3,912,232 after #703 data identical after #703

    How JFR found the Disco bug

    Winner events with segment = -1 are the global-dict admission probe. The Disco recording had none, while the other integer columns had one each:

    probe (segment -1) Winner events:  Disco 0   Hora_Pagina 3   Codigo 3
    Disco root winners: 12x fastlanes.bitpacked, 1x vortex.dict, 1x vortex.runend
    

    So Disco was ruled out before the probe ran. That pointed straight at the pre-probe filter, DictColumnState.admitsPtype, which admitted only I32/U32, I64/U64 and F64, while Rust's dict_layout_supported admits every primitive. The diffed trees agreed: Rust wrote one primitive(bool) values pool plus runend(for(bitpacked), bitpacked) codes, and Java wrote bitpacked(bool) per chunk. That is exactly the "Rust tree" / "Java tree" pair in this ticket's table.

    After #703 the probe appears, and vortex.dict wins on the stats estimate and on the measured encode:

    Column Segment Cascade Path  Depth Parent Scheme            Input     Compressed Estimated Achieved Accepted
    Disco       -1                   0        vortex.dict         17.0 kB   7.1 kB      2.538     2.387    true
    Disco       -1 dict.0            1 dict   fastlanes.bitpacked  9.0 kB   7.0 kB      1.286     1.286    true
    Disco       -1 dict.1            1 dict   fastlanes.for       124 B     896 B       2.286     0.138    false  (not smaller than the uncompressed array)
    Disco root winners after: 10x vortex.runend, 2x vortex.dict, 2x fastlanes.bitpacked
    

    The last row also shows what accepted = false is for: FoR on a 124-byte code array was estimated at 2.29x but measured at 0.14x (bit-packing padding), and the compressor kept the canonical array.

    What's left, and small: the zone-map statistics table of Hora_Pagina, Codigo and Disco is encoded slightly differently (Java bit-packs those min/max columns with patches, where Rust keeps vortex.dict over primitive). It amounts to bytes per file and is #583's territory.

  2. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Fixed in 2fcf17c. The remaining zone-table gap was the dictionary's value order: Rust's IntDictScheme/FloatDictScheme take the values in the iteration order of their hashbrown + FxHash distinct-values map, Java took them first-seen. Same values under other codes gave a different code bit-width histogram, so the codes bit-packed with patches (Disco) or bit-packed where Rust kept them primitive (Hora_Pagina). It shows only at 5M rows; at 2.4M the two orders happened to agree.

    DictValueOrder now replays that map (FxHasher, hashbrown 0.17 probing, growth and iteration; group width from os.arch, as Rust's SSE2/NEON). On the first 5M rows of Generico, Codigo, Disco and Hora_Pagina now write vortex-jni's encoding trees, data and zone table, in one batch or 64K batches. DictValueOrderInteropIntegrationTest compares every buffer byte for byte for the dense, hashed and float collectors, and fails without the fix.

  3. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Follow-ups after review, all on main:

    • 5e19246: an unsigned 8/16/32-bit dictionary whose values straddle the type's top bit took Rust's hashed counter instead of the dense one (Java's sign-extended bits). New byte-for-byte vortex-jni cross-checks for the corners that had none: that case, a float set filled exactly and grown by a duplicate, and integer keys piling up past 16 buckets (where hashbrown probes by group width, so each CI host checks its own).
    • 45eb71e + f08a388: the dict layout's codes width for 8-bit columns. Rust 0.91 (vortex-jni moved to 0.91.0 in a368b9d) gives U8 codes only to non-nullable 8-bit values and U16 to nullable ones and wider types; Java always wrote U16. 45eb71e applied 0.90's rule and was corrected by f08a388; DictLayoutInteropIntegrationTest now compares every narrow type, nullable and not, byte for byte.

    The write benchmark shows no regression from the dictionary ordering (javaWrite depth 3, 3 forks, interleaved: 0.825/0.768 ops/s before, 0.891/0.956 after).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions