You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries #649
Synthetic i32/u32/i16-range, mixed and full-i64 columns already produce identical trees in both writers, so narrowing is not involved.
Rust (vortex-layout/src/layouts/dict/writer.rs, DictStrategy): compresses a probe sample with the cascade; if the result is a Dict, it writes a dict layout, up to u16::MAX values per dictionary, and starts a new dictionary when the constraint is hit.
Java (DictColumnState.isDictCandidate): a numeric column is offered a global dict only with at most GLOBAL_DICT_MAX_CARDINALITY (2,048) distinct values and under 50% distinct, one dictionary for the whole column, demoted to per-chunk encoding past the cap. This column has ~23k distinct per chunk and ~719k over 2M rows.
Fix
Raising the cap is not enough: over the whole column Rust writes several dictionaries in sequence. Parity means replacing the cardinality heuristic with Rust's probe-then-multi-dictionary flow (touches ADR 0021, and the comment on the cap says a global dict hurt some high-cardinality I64/F64 columns, so check those for regressions).
Repro: Int64NarrowingInteropIntegrationTest (exploratory, not committed), fed HASH_CSV=<one-column dump>.
changed the title [-]Writer: 64-bit integer columns whose values fit 32 bits stay 8 bytes a value; Rust's are ~2x smaller[/-][+]Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries[/+]on Nov 2, 2026
Related: #679 added Rust's multi-dictionary flow for text columns (DictColumnState.ingestVarBinChunk: close the open dictionary at 65,535 entries / 1 MiB and start the next; the writer keeps a list of dictionaries per column and writes a chunked layout of dict layouts). Numeric columns still use the single dictionary with the 2,048 cap (ingestDictChunk), so this issue remains: apply the same run-based flow to primitives (Rust's primitive builder has the same max_len / max_bytes constraints in builders/dict/primitive.rs), then replace the cardinality heuristic with the probe, and re-check the high-cardinality I64/F64 columns the cap comment mentions.
Fixed by #682: numeric columns follow Rust's probe-then-multi-dictionary flow. Measured on the first 1M rows of ClickBench hits (RefererHash + URLHash): main 16,009,470 B vs vortex-jni 8,446,868 B (1.90x); with #682 8,445,894 B (1.00x). Not benchmarked: high-cardinality random I64/F64 columns (the probe decides them as in Rust; interop tests cover pools of 100 to 150,000 values).
Activity
[-]Writer: 64-bit integer columns whose values fit 32 bits stay 8 bytes a value; Rust's are ~2x smaller[/-][+]Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries[/+]dfa1 commentedon Nov 6, 2026
Related: #679 added Rust's multi-dictionary flow for text columns (
DictColumnState.ingestVarBinChunk: close the open dictionary at 65,535 entries / 1 MiB and start the next; the writer keeps a list of dictionaries per column and writes a chunked layout of dict layouts). Numeric columns still use the single dictionary with the 2,048 cap (ingestDictChunk), so this issue remains: apply the same run-based flow to primitives (Rust's primitive builder has the samemax_len/max_bytesconstraints inbuilders/dict/primitive.rs), then replace the cardinality heuristic with the probe, and re-check the high-cardinality I64/F64 columns the cap comment mentions.fix(writer): a numeric column keeps its dict layout past 2,048 distin…
dfa1 commentedon Nov 7, 2026
Fixed by #682: numeric columns follow Rust's probe-then-multi-dictionary flow. Measured on the first 1M rows of ClickBench hits (RefererHash + URLHash): main 16,009,470 B vs vortex-jni 8,446,868 B (1.90x); with #682 8,445,894 B (1.00x). Not benchmarked: high-cardinality random I64/F64 columns (the probe decides them as in Rust; interop tests cover pools of 100 to 150,000 values).