Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Breaking:** CSV import stores an empty cell in a number or boolean column as null instead of `0` or `false`, so inferred number and boolean columns are now nullable (read through `MaskedArray`), and an empty cell or a missing column fails the import where the schema declares the column non-nullable. ([#592](https://github.com/dfa1/vortex-java/issues/592))

### Fixed
- ALP writes patch values that are all equal as a `vortex.constant`, as Rust does. ([#619](https://github.com/dfa1/vortex-java/pull/619))
- Bit-packing writes its patches as Rust does: indices and per-1024-value chunk offsets narrowed to the smallest unsigned type that holds them, and equal values as a `vortex.constant`. ([#619](https://github.com/dfa1/vortex-java/pull/619))
- The cascade tries encodings in Rust's registration order, so a tie between two equally small encodings goes the way Rust breaks it. ([#604](https://github.com/dfa1/vortex-java/issues/604))
- ALP-RD packs its left and right parts at the dictionary's widths, as Rust does, instead of narrower widths that made it win where Rust picks a dictionary. ([#604](https://github.com/dfa1/vortex-java/issues/604))
- ALP on an `F32` column compresses its encoded integers through the cascade, as Rust does for both float widths, instead of storing them raw. ([#604](https://github.com/dfa1/vortex-java/issues/604))
Expand Down
2 changes: 1 addition & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.decimal_byte_parts` | `DecimalBytePartsEncodingDecoder`| `DecimalBytePartsEncodingEncoder`| ✅ | ✅ | Write: precision ≤ 18 only (one i64 mantissa, no low parts); competes in the cascade |
| `vortex.datetimeparts` | `DateTimePartsEncodingDecoder` | `DateTimePartsEncodingEncoder` | ✅ | ✅ | |
| `vortex.pco` | `PcoEncodingDecoder` | `PcoEncodingEncoder` | ✅ | ✅ | Decode: all modes. Encode: Classic + Consecutive (orders 1–7) delta + IntMult, in Rust's chunk/page layout and byte-identical to pco 1.0.1 for those; FloatMult/FloatQuant and Lookback deferred |
| `fastlanes.bitpacked` | `BitpackedEncodingDecoder` | `BitpackedEncodingEncoder` | ✅ | ✅ | Unsigned integer PTypes |
| `fastlanes.bitpacked` | `BitpackedEncodingDecoder` | `BitpackedEncodingEncoder` | ✅ | ✅ | Unsigned integer PTypes. Patches are written as Rust's `compress_patches` leaves them: narrowed indices, values as a `vortex.constant` when all equal, and per-1024-value chunk offsets |
| `fastlanes.delta` | `DeltaEncodingDecoder` | `DeltaEncodingEncoder` | ✅ | ✅ | Integer PTypes. In no edition: cascading writes offer it (bases/deltas cascaded, as Rust's `DeltaScheme`) only with `WriteOptions.withoutEditions()` |
| `fastlanes.for` | `FrameOfReferenceEncodingDecoder`| `FrameOfReferenceEncodingEncoder`| ✅ | ✅ | Integer PTypes |
| `fastlanes.rle` | `RleEncodingDecoder` | `RleEncodingEncoder` | ✅ | ✅ | Chunk-based RLE. Integers and floats (Rust's int and float RLE schemes); float runs compare raw bits, so -0.0 and NaN payloads round-trip. Cascades values/indices/offsets |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,16 @@ static Stream<Arguments> shapes() {
i -> (i / 20) % 7 == 0),
// taxi Airport_fee: one dominant value, two rare ones, nulls: a dict layout
shape("f64 airport fee", PType.F64, i -> i % 17 == 0 ? 1.75 : i % 101 == 0 ? -1.75 : 0.0,
i -> i % 10 == 3));
i -> i % 10 == 3),
// a few values ALP cannot round-trip, all equal: its patch values are a constant
shape("f64 equal exceptions", PType.F64, i -> i % 97 == 0 ? Math.PI : twoDecimals.applyAsDouble(i),
every7),
// the same in F32, over halves: ALP with constant patch values (two decimals
// would go to ALP-RD in F32), with and without nulls
shape("f32 equal exceptions", PType.F32, i -> i % 97 == 0 ? Math.PI : Math.floorMod(RANDOM[i], 1000) * 0.5,
every7),
shape("f32 equal exceptions, no nulls", PType.F32,
i -> i % 97 == 0 ? Math.PI : Math.floorMod(RANDOM[i], 1000) * 0.5, _ -> false));
}

@ParameterizedTest(name = "{0}")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,7 @@
///
/// Each column goes through vortex-jni and vortex-java, and the two files must hold the same
/// encoding tree, data and zone stats table alike. Each shape steers the cascade to a different
/// winner. Not covered: a column whose bit-packing needs patches, as Rust also stores the patches'
/// chunk offsets and Java does not yet.
/// winner.
class NullableIntegerInteropIntegrationTest {

private static final Session SESSION = Session.create();
Expand Down Expand Up @@ -97,7 +96,13 @@ static Stream<Arguments> shapes() {
shape("95% null", PType.I32, i -> Math.floorMod(RANDOM[i], 1000), i -> Math.floorMod(RANDOM[i] >>> 20, 100) < 95),
shape("clustered nulls", PType.I32, i -> Math.floorMod(RANDOM[i], 1000), i -> (i / 500) % 3 == 0),
// one null in one zone: that zone's null count makes the stats table's dict
shape("one null", PType.I32, i -> Math.floorMod(RANDOM[i], 1000), i -> i == 5));
shape("one null", PType.I32, i -> Math.floorMod(RANDOM[i], 1000), i -> i == 5),
// a few wide values: bit-packing with patches, which carry one offset per 1024 rows
shape("wide outliers", PType.I32, i -> i % 97 == 0 ? 1_000_000 + Math.floorMod(RANDOM[i], 1_000_000)
: Math.floorMod(RANDOM[i], 1000), every7),
// all outliers equal: the patch values are a constant
shape("equal outliers", PType.I32, i -> i % 97 == 0 ? 1_000_000_000 : Math.floorMod(RANDOM[i], 1000),
every7));
}

@ParameterizedTest(name = "{0}")
Expand Down
Loading
Loading