You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nyc-311: global-dict street columns ~54MB larger than vortex-jni (dict codes not bitpacked?) #303
After #299 (and PR #302, FSST child cascade), vortex-java is now 30 MB smaller than the vortex-jni reference on nyc-311 (1731 vs 1762 MB). The largest remaining cluster where java is bigger is three high-cardinality street columns — pure upside to close.
~54 MB combined. All three are now admitted to the global dictionary (the #299 type-aware Utf8 cap of 32768 works — they round-trip as vortex.dict layouts), and their values pool is FSST-compressed (#299 Finding 2). But jni's layout is ~36% smaller.
Hypothesis
Two likely, non-exclusive causes — both worth checking before fixing:
Dict codes stored raw, not bitpacked. The global-dict codes are per-chunk U16 arrays (18.49M rows × 2 B ≈ 37 MB just for codes). jni tags these columns fastlanes.bitpacked. A dict of ~27k values needs only 15 bits, and finer per-chunk cardinality needs fewer — so bitpacking (or FoR) the codes could recover several MB per column. This is the same class of issue PR Cascade FSST offset/length children (#299) #302 fixed for the FSST offset/length children: a child left as raw vortex.primitive instead of cascaded. Check whether writeGlobalDictUtf8Column's code segments actually bitpack (they route through writeSegment → cascade; a nullable-masked wrapper may be blocking the inner bitpack).
Global vs per-chunk dict tradeoff. jni appears to use a per-chunk dict (dict encoding, not a global dict layout) with its finer 32k–64k chunking, so each chunk's code width tracks that chunk's (smaller) local cardinality rather than the global 27k. At this cardinality the per-chunk approach may simply win.
Confirmed the root cause by dumping the encoding trees for Cross Street 1 in both files:
vortex-jni (32 MB) vortex-java (50 MB)
DICT (one global dict) CHUNKED (283 per-chunk dicts)
values: vortex.fsst per chunk:
codes: fastlanes.bitpacked values: vortex.fsst (repeated ×283)
codes: vortex.primitive ← RAW
Both findings from the issue are real:
Finding 1 — codes not bitpacked → fixed in #305.encodeUtf8Cascade exposed the values pool as an open cascade child but left the codes as a raw vortex.primitive leaf. Exposing codes as an open slot too lets them bitpack. Measured: nyc-311 1934 → 1862 MB (−72 MB), helping every per-chunk Utf8 dict column.
Finding 2 — per-chunk vs global dict (still open). These columns fall back to a per-chunk dict (283 chunks, dictionary repeated each time) instead of one global dict, even though their cardinality (~20–28k distinct) is under the 32768 global cap. Likely retained-budget eviction demotes them. Keeping them global would store the values pool once instead of 283×. Leaving this issue open to track it.
Activity
dfa1 commentedon Jul 25, 2026
Confirmed the root cause by dumping the encoding trees for Cross Street 1 in both files:
Both findings from the issue are real:
Finding 1 — codes not bitpacked → fixed in #305.
encodeUtf8Cascadeexposed the values pool as an open cascade child but left the codes as a rawvortex.primitiveleaf. Exposing codes as an open slot too lets them bitpack. Measured: nyc-311 1934 → 1862 MB (−72 MB), helping every per-chunk Utf8 dict column.Finding 2 — per-chunk vs global dict (still open). These columns fall back to a per-chunk dict (283 chunks, dictionary repeated each time) instead of one global dict, even though their cardinality (~20–28k distinct) is under the 32768 global cap. Likely retained-budget eviction demotes them. Keeping them global would store the values pool once instead of 283×. Leaving this issue open to track it.
feat(writer): cascade Utf8 dict codes so they bitpack (#303)
perf(writer): raise default global-dict retained budget 1GB -> 2GB (#303