You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A string column whose first 8192 rows are constant gets a dict layout Rust does not write #722
Rust's dict layout runs its probe compressor over the first 8192 rows and takes the dict layout only when the result is a Dict (vortex-layout/src/layouts/dict/writer.rs:169-182, 0.91). A constant sample compresses to Constant, so Rust falls back to plain chunked flats.
For Utf8/Binary, GlobalDictWriter.detect uses DictColumnState.isVarBinDictCandidate ("not more than half distinct") instead, which admits a sample of 0 or 1 distinct values.
Found on Raincloud coig: the first 8192 rows of instruction are all "", so Java writes a global dict layout and the column is 8.8% bigger than Rust's (5.27 MB vs 4.85 MB from vortex-jni 0.91). With withGlobalDict(false) Java writes 4.84 MB in 13 chunks, Rust's size and chunking. The constant split columns of coig and mnist get a dict layout for the same reason.
Fix: a sample with at most one distinct non-null value is not a candidate. The full parity fix is to ask the cascade probe for strings too, as primitives do, once the string scheme picks Dict where Rust's does (it picks OnPair today).
Test first: a vortex-jni cross-check on a string column whose first 8192 rows are constant and the rest varied, asserting both writers pick the same layout.
Activity