Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `CompressorBuilder#seed(long)` of the `vortex-fsst` module: training takes Rust's fixed sample, not a seeded one. ([#583](https://github.com/dfa1/vortex-java/issues/583))

### Fixed
- FSST symbol tables are trained as Rust's `fsst-rs` does (its sample, generations, candidate gains and cut), so a text column without a null gets Rust's FSST encoding up to ties of equal gain, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- FSST symbol tables are trained as Rust's `fsst-rs` 0.7 does, so a text column without a null gets Rust's FSST encoding byte for byte, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- `VortexReader.columnStats()` reports the min and max of a decimal column, and a decimal filter compares by value instead of by the long it truncates to. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650))
- 8-bit, 16-bit and `F32` columns get Rust's dict layout when its probe picks one, so a nullable low-cardinality `SMALLINT` is written up to 30% smaller. ([#651](https://github.com/dfa1/vortex-java/issues/651))
- A filtered read of a decimal column whose flat node carries min/max stats no longer fails with `VortexException`; those bounds are not used for pruning. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696))
- **Breaking:** a `Double`, `Float` or `BigDecimal` filter operand on an integer column fails with `VortexException`, as vortex-jni does, instead of being truncated so that `x >= 1.5` matched `x = 1`. ([#701](https://github.com/dfa1/vortex-java/issues/701))
- A `vortex.variant` column whose shredded dtype has a control character in a field or variant name fails with `VortexException` instead of `IllegalArgumentException`. ([#694](https://github.com/dfa1/vortex-java/issues/694))
- Written files carry the per-column file statistics (min, max, sum, null and NaN counts) in the footer, as vortex-jni writes them. ([#664](https://github.com/dfa1/vortex-java/issues/664))
- Written files carry the per-column file statistics (min, max, null and NaN counts, uncompressed size) in the footer, as vortex-jni 0.91 writes them. ([#664](https://github.com/dfa1/vortex-java/issues/664))
- A column written as one chunk has no `vortex.chunked` layout around it, as Rust writes it. ([#711](https://github.com/dfa1/vortex-java/pull/711))
- An integer or float dictionary inside the cascade, such as a zone table's min column, orders its values as Rust does, so its codes and their encoding are Rust's. ([#651](https://github.com/dfa1/vortex-java/issues/651))

Expand Down
8 changes: 4 additions & 4 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Compatibility

Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.90.0.
Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.91.0.
For the rest of the API surface (reader, writer, scan, CLI), see [reference.md](reference.md).

## Read-only deployment
Expand Down Expand Up @@ -42,7 +42,7 @@ only the built-in decoders in `reader`; no encoder class is loaded.
| Arrow extension array import affecting Variant shape | Rust 0.74.0 (#8125) | Untested against the currently pinned v0.88.0 fixtures; #8125 not yet re-verified. |
| `vortex.dict` **layout** over a values pool that is neither VarBin- nor primitive-shaped (e.g. a dict-encoded `vortex.uuid`, whose storage is `FixedSizeList(U8, 16)`) | Not written by Rust: its dict layout admits only `Primitive \| Utf8 \| Binary` (`dict_layout_supported`), and its dict compressor schemes only integers, floats and strings | ⚠️ Unreachable from Rust- or Java-written files. Every type vortex-jni writes reads back exactly, dict-encoded or not (`DictAllTypesInteropIntegrationTest`, which also fails if a Rust bump starts dict-encoding another type). A foreign file with such a pool fails with `VortexException("unsupported dict values shape: …")`. |
| Duplicate struct field names | Rust writer rejects ("StructLayout must have unique field names"); Rust reader tolerates foreign files (first-match access) | ⚠️ Deliberate divergence: the `DType.Struct` constructor rejects duplicate names at every nesting level with `VortexException("duplicate field name")`, where Rust's in-memory `StructFields` tolerates them. So Java can neither build such a schema to write nor read such a file — the name-keyed `Chunk` API cannot represent both columns, and silent column loss is worse than a loud failure on a file the reference writer refuses to produce. |
| Decimal values beyond their declared precision | Wire-legal (a decimal array stores whatever its unscaled integers hold). Rust 0.90's Arrow import keeps such values; Rust's development branch rejects them ([vortex-data/vortex#10897](https://github.com/vortex-data/vortex/pull/10897): "Arrow decimal import requires every valid value to be within the precision") | ⚠️ Deliberate strictness on write: vortex-java's writer refuses a value with more digits than its column's precision, or more fractional digits than its scale (`IllegalArgumentException` naming the column and row, pinned by `ChunkImplTest`), whether the value comes through the API or a Parquet import. A value outside its declared type is wrong data, not a wider decimal. |
| Decimal values beyond their declared precision | Wire-legal (a decimal array stores whatever its unscaled integers hold). Rust's Arrow import rejects them from 0.91 ([vortex-data/vortex#10897](https://github.com/vortex-data/vortex/pull/10897): "Arrow decimal import requires every valid value to be within the precision"); 0.90 kept them | ✅ Rust parity from 0.91: vortex-java's writer refuses a value with more digits than its column's precision, or more fractional digits than its scale (`IllegalArgumentException` naming the column and row, pinned by `ChunkImplTest`), whether the value comes through the API or a Parquet import. A value outside its declared type is wrong data, not a wider decimal. |
| Blank / control-character field names | Wire-legal; the Rust writer produces `""` and whitespace-only names. NUL (`U+0000`) additionally aborts the Rust toolchain: Arrow FFI schema export hits a panic-cannot-unwind in `arrow-rs` (`ffi_stream::get_schema`) and SIGABRTs the process (measured against vortex-jni 0.75.0) | ⚠️ Deliberate strictness BOTH ways: vortex-java's writer refuses blank and control-character field names (`IllegalArgumentException`), and its reader rejects files carrying them (`VortexException` naming the producing pipeline as the likely bug) — the JSON-`""`-key principle: wire-legal is a floor, not a policy. Printable names of any shape (`$`-runs, spaces inside, emoji) are legal and round-trip intact both directions (measured; pinned by `ColumnNameEdgeCasesIntegrationTest`). |

## SIMD kernels and the Vector API
Expand Down Expand Up @@ -134,7 +134,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) |
| `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct |
| `vortex.chunked` | `ChunkedEncodingDecoder` | `ChunkedEncodingEncoder` | ✅ | ✅ | Primitive + Struct concat |
| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary. The symbol table is trained as Rust's `fsst-rs` v0.6.0 does ([#583](https://github.com/dfa1/vortex-java/issues/583)): its sample, five generations, candidate gains, cut and final code order. Not byte for byte Rust's, by decision: candidates of equal gain and length pop in the order they were counted, where Rust pops them in its hash map's iteration order, which depends on the CPU's SIMD width (Rust's own table differs between x86 and arm); about 90% of the symbols and a code stream within 1% are Rust's. A column with nulls trains on its empty placeholder rows, where Rust trains on the valid rows only, until its validity lives in the encoding (#650) |
| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary. The symbol table is the one Rust's `fsst-rs` 0.7 trains, byte for byte ([#583](https://github.com/dfa1/vortex-java/issues/583)): its sample, five generations, candidate gains, cut, tie order and final code order. A column with nulls trains on its empty placeholder rows, where Rust trains on the valid rows only, until its validity lives in the encoding (#650) |
| `vortex.list` | `ListEncodingDecoder` | `ListEncodingEncoder` | ✅ | ✅ | A nullable list's validity is its optional third child, as in Rust |
| `vortex.listview` | `ListViewEncodingDecoder` | `ListViewEncodingEncoder` | ✅ | ✅ | Validity lives in the encoding's own fourth child slot, not under a `vortex.masked` wrapper. Read: both shapes decode. Write: only `vortex.map`'s entries child takes this path — a plain nullable `DType.List` column still routes through `vortex.masked` + `vortex.list` |
| `vortex.map` | `MapEncodingDecoder` | `MapEncodingEncoder` | ✅ | ✅ | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately |
Expand Down Expand Up @@ -290,7 +290,7 @@ guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of
| `preview2026.08.0` | `preview` | nothing yet |
| `zstd2026.02.0` | `zstd` | `vortex.zstd_buffers` (read only; buffer-level Zstd for GPU decode; declared by Rust's `vortex-zstd` plugin, opt-in) |

Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.90.0);
Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.91.0);
`EditionCatalogParityIntegrationTest` fails the build on drift. `core` editions are frozen with a
forever read-compatibility guarantee; `preview` holds opt-in components not yet adopted by `core`;
`zstd` is declared by Rust's `vortex-zstd` plugin crate rather than the core declarations, and is
Expand Down
4 changes: 2 additions & 2 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ Writes a Vortex file. Implements `Closeable`. The file is complete and readable
| `static create(WritableByteChannel, DType.Struct, WriteOptions, WriteRegistry)` | Custom [`WriteRegistry`](#writeregistry-iogithubdfa1vortexwriterwriteregistry) — encoders and extension encoders together |
| `writeChunk(Consumer<Chunk>)` | One batch of rows; typed builder validates column names + array types at each `.put`; missing columns throw `IllegalStateException` when the lambda returns. Preferred when columns are known at compile time. |
| `writeChunk(Map<ColumnName, Object>)` | One batch of rows by map. Validates that every schema column is present and that all columns share the same row count. Use when the column set is built dynamically (Parquet/JDBC importers, generic exporters). |
| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, sum, null and NaN counts, folded over the batches, as vortex-jni's `VortexWriter.finish()` reports them). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` |
| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, null and NaN counts and uncompressed size, accumulated over the batches, as vortex-jni's `VortexWriter.finish()` reports them; a UTF-8 or binary min/max is a 64-byte bound, never exact). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` |
| `close()` | `finish()` unless it already ran |

Batches are not chunks: as Rust's writer does, each primitive, boolean, decimal, string and binary
Expand Down Expand Up @@ -236,7 +236,7 @@ returned. The filter's columns are decoded even when they are not projected.

An integer column takes an integer operand of any width (`Long`, `Integer`, `Short`, `Byte`). A
`Double`, `Float` or `BigDecimal` operand on an integer column fails the scan with
`VortexException`, as vortex-jni refuses it ("Cannot compare different DTypes i64 and f64").
`VortexException`, as vortex-jni refuses it ("Comparison requires equal dtypes ignoring nullability, got i64 and f64").

| Static factory | Builds |
|------------------------------|------------------------------------|
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,12 @@

import java.util.List;

/// Trains an FSST [Compressor] from a corpus of rows as Rust's `fsst-rs` does (`Compressor::train`,
/// Trains an FSST [Compressor] from a corpus of rows as Rust's `fsst-rs` 0.7 does (`Compressor::train`,
/// the FSST paper's Algorithm 3 with the C++ reference's refinements): the same sample, the same five
/// generations over a growing share of its lines, and the same candidates, gains and cut.
///
/// Training is deterministic: the same rows always produce a byte-identical [Compressor]. Rust's own
/// tables differ from this one only in where candidates of exactly equal gain and length are
/// ordered, which depends on its hash map ([TrainingGeneration]).
/// Training is deterministic: the same rows always produce a byte-identical [Compressor], the table
/// fsst-rs 0.7 trains from them ([TrainingGeneration]).
///
/// ```java
/// Compressor compressor = new CompressorBuilder().train(rows);
Expand Down
Loading
Loading