Skip to content

Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries #649

Closed

Description

@dfa1

Raincloud clickbench-hits, vortex-java's Parquet import vs the vortex-jni-written file, data bytes (99.997M rows):

column type Rust Java
RefererHash I64 401.0 MB 791.3 MB (1.97x)
URLHash I64 427.3 MB 790.0 MB (1.85x)

Cause (confirmed)

Not integer narrowing: the values are full 64-bit. Rust writes these columns as a dict layout; Java writes plain vortex.primitive (8 bytes a row).

Reproduced with 131,072 real RefererHash rows (first chunk of ClickBench hits.parquet, ~23k distinct), written by both writers:

file tree
vortex-jni 434 KB vortex.dict layout: vortex.primitive values + fastlanes.bitpacked codes
vortex-java 1,050 KB vortex.primitive

Synthetic i32/u32/i16-range, mixed and full-i64 columns already produce identical trees in both writers, so narrowing is not involved.

  • Rust (vortex-layout/src/layouts/dict/writer.rs, DictStrategy): compresses a probe sample with the cascade; if the result is a Dict, it writes a dict layout, up to u16::MAX values per dictionary, and starts a new dictionary when the constraint is hit.
  • Java (DictColumnState.isDictCandidate): a numeric column is offered a global dict only with at most GLOBAL_DICT_MAX_CARDINALITY (2,048) distinct values and under 50% distinct, one dictionary for the whole column, demoted to per-chunk encoding past the cap. This column has ~23k distinct per chunk and ~719k over 2M rows.

Fix

Raising the cap is not enough: over the whole column Rust writes several dictionaries in sequence. Parity means replacing the cardinality heuristic with Rust's probe-then-multi-dictionary flow (touches ADR 0021, and the comment on the cap says a global dict hurt some high-cardinality I64/F64 columns, so check those for regressions).

Repro: Int64NarrowingInteropIntegrationTest (exploratory, not committed), fed HASH_CSV=<one-column dump>.

Activity

  1. changed the title [-]Writer: 64-bit integer columns whose values fit 32 bits stay 8 bytes a value; Rust's are ~2x smaller[/-] [+]Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries[/+] on Nov 2, 2026
  2. dfa1 commented on Nov 6, 2026

    @dfa1
    OwnerAuthor

    Related: #679 added Rust's multi-dictionary flow for text columns (DictColumnState.ingestVarBinChunk: close the open dictionary at 65,535 entries / 1 MiB and start the next; the writer keeps a list of dictionaries per column and writes a chunked layout of dict layouts). Numeric columns still use the single dictionary with the 2,048 cap (ingestDictChunk), so this issue remains: apply the same run-based flow to primitives (Rust's primitive builder has the same max_len / max_bytes constraints in builders/dict/primitive.rs), then replace the cardinality heuristic with the probe, and re-check the high-cardinality I64/F64 columns the cap comment mentions.

  3. dfa1 commented on Nov 7, 2026

    @dfa1
    OwnerAuthor

    Fixed by #682: numeric columns follow Rust's probe-then-multi-dictionary flow. Measured on the first 1M rows of ClickBench hits (RefererHash + URLHash): main 16,009,470 B vs vortex-jni 8,446,868 B (1.90x); with #682 8,445,894 B (1.00x). Not benchmarked: high-cardinality random I64/F64 columns (the probe decides them as in Rust; interop tests cover pools of 100 to 150,000 values).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions