You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
F32 two-decimal floats compress to 104% of raw (vortex-jni: 22%); F64 skips bitpacking under ALP #458
Low-precision floats, such as probabilities rounded to two decimals, compress far worse with vortex-java than with vortex-jni on the same data. For F32 the output is larger than raw.
Data
These are the 1,124 probabilities in the cached jev-1.13.0 answers (jev4j src/test/resources/jev/*.json: yes/no scores, confidences, choice and score distributions).
Every value has two decimals in [0, 1]. Only 99 distinct values occur; 39% are 0.0 and 10% are 1.0.
Three values are float artifacts (0.5700000000000001, 0.9299999999999999).
Shannon entropy is about 4.4 bits/value.
Repro: draw 1M rows at random from those values (new Random(42)), as a single non-nullable column p, written in 65,536-row batches with WriteOptions.defaults() and with vortex-jni 0.86.1 (Arrow Float4/Float8).
4.18 MB: vortex.alp with its i32 integers stored as plain vortex.primitive (~33 bits/value)
Expected
Pick encodings the way Rust does: replicate the vortex-jni choices by diffing the two encoding trees on the same input.
F32: ALP's encoded i32 child must cascade (narrowing/bitpacking) as the F64 path partly does, and vortex.dict should be a candidate where Rust's float dict scheme wins.
F64: the ALP integers should be bitpacked, as Rust does.
c72dc974 makes WriteOptions.defaults() cascade, as Rust's writer does. Default F32 output for this data drops from 4.42 MB to 931 KB, picking dict → bitpacked, the same choice as vortex-jni (923 KB on the same synthetic data). F64 goes from 1.06 MB to 924 KB (vortex-jni: 938 KB).
Remaining gap: branch wip/458-f32-alp-cascade makes ALP's F32 ints and dict values cascade, matching Rust's schemes. On its own it makes F32 larger (1.14 MB), because Java compresses each 65,536-row batch while Rust re-chunks into ~1 MB blocks (262,144 F32 rows). On the smaller sample ALP beats dict. So this needs Rust's coalescing repartition first, which in turn needs Writer emits legacy vortex.stats layout; Rust's default writer emits vortex.zoned #447's fixed 8192-row zones.
On main: default writes cascade (c72dc974) and emit Rust's vortex.zoned with 8192-row zones (8f867f91, #447). Default F32 for this data is now 931 KB, using dict → bitpacked as vortex-jni does (vortex-jni: 923 KB). F64 is 924 KB (vortex-jni: 938 KB).
What's left, on branches:
wip/repartition: coalesces batches into ~1 MB chunks, as Rust's RepartitionStrategy does. It breaks ~51 writer tests (plus more in Calcite and integration) that assume one chunk per writeChunk batch, and Rust has no setting that preserves that.
wip/458-f32-alp-cascade: ALP F32 ints and dict values cascade. Only worthwhile on top of the repartition.
Decision: not worth a large test rewrite or a Java-only chunking option for ~1% on F32.
Not merged: wip/repartition (Rust's ~1 MB re-chunking) and wip/458-f32-alp-cascade. Together they're worth about 1%, and re-chunking would break the one-chunk-per-writeChunk-batch behavior that many tests rely on.
Activity
feat(writer)!: cascade by default, as Rust's writer does
dfa1 commentedon Oct 8, 2026
Status:
c72dc974makesWriteOptions.defaults()cascade, as Rust's writer does. Default F32 output for this data drops from 4.42 MB to 931 KB, picking dict → bitpacked, the same choice as vortex-jni (923 KB on the same synthetic data). F64 goes from 1.06 MB to 924 KB (vortex-jni: 938 KB).wip/458-f32-alp-cascademakes ALP's F32 ints and dict values cascade, matching Rust's schemes. On its own it makes F32 larger (1.14 MB), because Java compresses each 65,536-row batch while Rust re-chunks into ~1 MB blocks (262,144 F32 rows). On the smaller sample ALP beats dict. So this needs Rust's coalescing repartition first, which in turn needs Writer emits legacy vortex.stats layout; Rust's default writer emits vortex.zoned #447's fixed 8192-row zones.fix: compare strings in UTF-8 byte order; skip NaN in float min/max
dfa1 commentedon Oct 8, 2026
Parking the remaining ~1% here.
On main: default writes cascade (
c72dc974) and emit Rust'svortex.zonedwith 8192-row zones (8f867f91, #447). Default F32 for this data is now 931 KB, using dict → bitpacked as vortex-jni does (vortex-jni: 923 KB). F64 is 924 KB (vortex-jni: 938 KB).What's left, on branches:
wip/repartition: coalesces batches into ~1 MB chunks, as Rust'sRepartitionStrategydoes. It breaks ~51 writer tests (plus more in Calcite and integration) that assume one chunk perwriteChunkbatch, and Rust has no setting that preserves that.wip/458-f32-alp-cascade: ALP F32 ints and dict values cascade. Only worthwhile on top of the repartition.Decision: not worth a large test rewrite or a Java-only chunking option for ~1% on F32.
fix(writer): size cascade samples as Rust does, a multiple of 1024 rows
wip(writer): cascade F32 ALP ints and dict values, as Rust (#458)
dfa1 commentedon Oct 8, 2026
Closing: default writes now match vortex-jni's encoding choices for this data.
c72dc974: writes cascade by default, as Rust's writer does.215bdcb3: cascade samples are sized as Rust sizes them (a multiple of 1024 rows), so bit-packing is no longer under-measured on large batches.Results on 1M rows:
Not merged:
wip/repartition(Rust's ~1 MB re-chunking) andwip/458-f32-alp-cascade. Together they're worth about 1%, and re-chunking would break the one-chunk-per-writeChunk-batch behavior that many tests rely on.docs: refresh benchmarks and status after the cascade parity work