Description
Problem
OnPairEncodingEncoder does not draw the same training sample as the Rust onpair crate, so it trains different dictionaries on identical data. Prime directive: behavioral parity with Rust.
Found by comparing a 100k-row URL column (https://example.com/products/<n>/reviews?page=<m>, 7 chunks of 16,384 rows) written by vortex-java and by Rust 0.88.0 (the release vortex-jni uses), both at cascade 3 with global dict off. Both pick vortex.onpair per chunk.
dict_bytes_len per chunk:
| chunk | Rust | Java |
|---|---|---|
| 0 | 8191 | 8429 |
| 1 | 8525 | 8368 |
| 2 | 8447 | 8387 |
| 3 | 8363 | 8452 |
| 4 | 8379 | 8473 |
| 5 | 8757 | 5670 |
| 6 | 1872 | 1877 |
Chunks 0-4 end up within 0.04% in total, but Java's chunk 5 is 18% smaller (131,904 B vs 161,507 B): its trainer built a 761-token dictionary and the cascade then dict-encoded the 467 distinct codes. That one chunk is 29.6 KB of the 31.6 KB (3.2%) by which Java's file is smaller. This is training luck, not a systematic advantage, and it could go the other way on other data.
Cause
- Rust (
onpair0.2.1,training_orderinencoding/trainer.rs):StdRng::seed_from_u64(42)(ChaCha12,rand0.9.5) andorder.partial_shuffle(&mut rng, k)withk = min(n, 2 * sample_fraction * n + 1024). Training consumes only the selected tailorder[n-k..]; if that tail is shorter than the byte budget,kdoubles and the shuffle restarts. - Java (
OnPairEncodingEncoder.Encoder.train):SplittableRandom(42), a full Fisher-Yates shuffle, and training consumesorderfrom the start.
Proposed fix
- Port
rand0.9.5partial_shuffleon top ofSampleRng(already a bit-exact ChaCha12StdRngandrandom_rangeforu32). It usesIncreasingUniform/calculate_bound_u32, which batches several swaps into one random chunk. Check thatSampleRng.nextIntInclusive(written againstrand0.10) gives the same values as 0.9.5'srandom_range. - Reproduce the
kcomputation, the tail consumption and the doubling retry loop intrain. - Parity test: the URL column above, Rust as the oracle, comparing
dict_bytes_lenand encoded size per chunk (expect identical).
Repro
Scratch crate with vortex = "=0.88.0"; data dumped from the Java CascadeDepthGainIntegrationTest patterns (utf8-urls), seed 42. Not committed.
Activity