diff --git a/CHANGELOG.md b/CHANGELOG.md index 67b22dd13..4c64cad18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `CompressorBuilder#seed(long)` of the `vortex-fsst` module: training takes Rust's fixed sample, not a seeded one. ([#583](https://github.com/dfa1/vortex-java/issues/583)) ### Fixed -- FSST symbol tables are trained as Rust's `fsst-rs` does (its sample, generations, candidate gains and cut), so a text column without a null gets Rust's FSST encoding up to ties of equal gain, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583)) +- FSST symbol tables are trained as Rust's `fsst-rs` 0.7 does, so a text column without a null gets Rust's FSST encoding byte for byte, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - `VortexReader.columnStats()` reports the min and max of a decimal column, and a decimal filter compares by value instead of by the long it truncates to. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650)) - 8-bit, 16-bit and `F32` columns get Rust's dict layout when its probe picks one, so a nullable low-cardinality `SMALLINT` is written up to 30% smaller. ([#651](https://github.com/dfa1/vortex-java/issues/651)) @@ -24,7 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696)) - **Breaking:** a `Double`, `Float` or `BigDecimal` filter operand on an integer column fails with `VortexException`, as vortex-jni does, instead of being truncated so that `x >= 1.5` matched `x = 1`. ([#701](https://github.com/dfa1/vortex-java/issues/701)) - A `vortex.variant` column whose shredded dtype has a control character in a field or variant name fails with `VortexException` instead of `IllegalArgumentException`. ([#694](https://github.com/dfa1/vortex-java/issues/694)) -- Written files carry the per-column file statistics (min, max, sum, null and NaN counts) in the footer, as vortex-jni writes them. ([#664](https://github.com/dfa1/vortex-java/issues/664)) +- Written files carry the per-column file statistics (min, max, null and NaN counts, uncompressed size) in the footer, as vortex-jni 0.91 writes them. ([#664](https://github.com/dfa1/vortex-java/issues/664)) - A column written as one chunk has no `vortex.chunked` layout around it, as Rust writes it. ([#711](https://github.com/dfa1/vortex-java/pull/711)) - An integer or float dictionary inside the cascade, such as a zone table's min column, orders its values as Rust does, so its codes and their encoding are Rust's. ([#651](https://github.com/dfa1/vortex-java/issues/651)) diff --git a/docs/compatibility.md b/docs/compatibility.md index 8c2fe4226..c626f3922 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -1,6 +1,6 @@ # Compatibility -Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.90.0. +Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.91.0. For the rest of the API surface (reader, writer, scan, CLI), see [reference.md](reference.md). ## Read-only deployment @@ -42,7 +42,7 @@ only the built-in decoders in `reader`; no encoder class is loaded. | Arrow extension array import affecting Variant shape | Rust 0.74.0 (#8125) | Untested against the currently pinned v0.88.0 fixtures; #8125 not yet re-verified. | | `vortex.dict` **layout** over a values pool that is neither VarBin- nor primitive-shaped (e.g. a dict-encoded `vortex.uuid`, whose storage is `FixedSizeList(U8, 16)`) | Not written by Rust: its dict layout admits only `Primitive \| Utf8 \| Binary` (`dict_layout_supported`), and its dict compressor schemes only integers, floats and strings | ⚠️ Unreachable from Rust- or Java-written files. Every type vortex-jni writes reads back exactly, dict-encoded or not (`DictAllTypesInteropIntegrationTest`, which also fails if a Rust bump starts dict-encoding another type). A foreign file with such a pool fails with `VortexException("unsupported dict values shape: …")`. | | Duplicate struct field names | Rust writer rejects ("StructLayout must have unique field names"); Rust reader tolerates foreign files (first-match access) | ⚠️ Deliberate divergence: the `DType.Struct` constructor rejects duplicate names at every nesting level with `VortexException("duplicate field name")`, where Rust's in-memory `StructFields` tolerates them. So Java can neither build such a schema to write nor read such a file — the name-keyed `Chunk` API cannot represent both columns, and silent column loss is worse than a loud failure on a file the reference writer refuses to produce. | -| Decimal values beyond their declared precision | Wire-legal (a decimal array stores whatever its unscaled integers hold). Rust 0.90's Arrow import keeps such values; Rust's development branch rejects them ([vortex-data/vortex#10897](https://github.com/vortex-data/vortex/pull/10897): "Arrow decimal import requires every valid value to be within the precision") | ⚠️ Deliberate strictness on write: vortex-java's writer refuses a value with more digits than its column's precision, or more fractional digits than its scale (`IllegalArgumentException` naming the column and row, pinned by `ChunkImplTest`), whether the value comes through the API or a Parquet import. A value outside its declared type is wrong data, not a wider decimal. | +| Decimal values beyond their declared precision | Wire-legal (a decimal array stores whatever its unscaled integers hold). Rust's Arrow import rejects them from 0.91 ([vortex-data/vortex#10897](https://github.com/vortex-data/vortex/pull/10897): "Arrow decimal import requires every valid value to be within the precision"); 0.90 kept them | ✅ Rust parity from 0.91: vortex-java's writer refuses a value with more digits than its column's precision, or more fractional digits than its scale (`IllegalArgumentException` naming the column and row, pinned by `ChunkImplTest`), whether the value comes through the API or a Parquet import. A value outside its declared type is wrong data, not a wider decimal. | | Blank / control-character field names | Wire-legal; the Rust writer produces `""` and whitespace-only names. NUL (`U+0000`) additionally aborts the Rust toolchain: Arrow FFI schema export hits a panic-cannot-unwind in `arrow-rs` (`ffi_stream::get_schema`) and SIGABRTs the process (measured against vortex-jni 0.75.0) | ⚠️ Deliberate strictness BOTH ways: vortex-java's writer refuses blank and control-character field names (`IllegalArgumentException`), and its reader rejects files carrying them (`VortexException` naming the producing pipeline as the likely bug) — the JSON-`""`-key principle: wire-legal is a floor, not a policy. Printable names of any shape (`$`-runs, spaces inside, emoji) are legal and round-trip intact both directions (measured; pinned by `ColumnNameEdgeCasesIntegrationTest`). | ## SIMD kernels and the Vector API @@ -134,7 +134,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul | `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) | | `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct | | `vortex.chunked` | `ChunkedEncodingDecoder` | `ChunkedEncodingEncoder` | ✅ | ✅ | Primitive + Struct concat | -| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary. The symbol table is trained as Rust's `fsst-rs` v0.6.0 does ([#583](https://github.com/dfa1/vortex-java/issues/583)): its sample, five generations, candidate gains, cut and final code order. Not byte for byte Rust's, by decision: candidates of equal gain and length pop in the order they were counted, where Rust pops them in its hash map's iteration order, which depends on the CPU's SIMD width (Rust's own table differs between x86 and arm); about 90% of the symbols and a code stream within 1% are Rust's. A column with nulls trains on its empty placeholder rows, where Rust trains on the valid rows only, until its validity lives in the encoding (#650) | +| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary. The symbol table is the one Rust's `fsst-rs` 0.7 trains, byte for byte ([#583](https://github.com/dfa1/vortex-java/issues/583)): its sample, five generations, candidate gains, cut, tie order and final code order. A column with nulls trains on its empty placeholder rows, where Rust trains on the valid rows only, until its validity lives in the encoding (#650) | | `vortex.list` | `ListEncodingDecoder` | `ListEncodingEncoder` | ✅ | ✅ | A nullable list's validity is its optional third child, as in Rust | | `vortex.listview` | `ListViewEncodingDecoder` | `ListViewEncodingEncoder` | ✅ | ✅ | Validity lives in the encoding's own fourth child slot, not under a `vortex.masked` wrapper. Read: both shapes decode. Write: only `vortex.map`'s entries child takes this path — a plain nullable `DType.List` column still routes through `vortex.masked` + `vortex.list` | | `vortex.map` | `MapEncodingDecoder` | `MapEncodingEncoder` | ✅ | ✅ | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately | @@ -290,7 +290,7 @@ guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of | `preview2026.08.0` | `preview` | nothing yet | | `zstd2026.02.0` | `zstd` | `vortex.zstd_buffers` (read only; buffer-level Zstd for GPU decode; declared by Rust's `vortex-zstd` plugin, opt-in) | -Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.90.0); +Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.91.0); `EditionCatalogParityIntegrationTest` fails the build on drift. `core` editions are frozen with a forever read-compatibility guarantee; `preview` holds opt-in components not yet adopted by `core`; `zstd` is declared by Rust's `vortex-zstd` plugin crate rather than the core declarations, and is diff --git a/docs/reference.md b/docs/reference.md index 3ac5020fa..540743121 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -145,7 +145,7 @@ Writes a Vortex file. Implements `Closeable`. The file is complete and readable | `static create(WritableByteChannel, DType.Struct, WriteOptions, WriteRegistry)` | Custom [`WriteRegistry`](#writeregistry-iogithubdfa1vortexwriterwriteregistry) — encoders and extension encoders together | | `writeChunk(Consumer)` | One batch of rows; typed builder validates column names + array types at each `.put`; missing columns throw `IllegalStateException` when the lambda returns. Preferred when columns are known at compile time. | | `writeChunk(Map)` | One batch of rows by map. Validates that every schema column is present and that all columns share the same row count. Use when the column set is built dynamically (Parquet/JDBC importers, generic exporters). | -| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, sum, null and NaN counts, folded over the batches, as vortex-jni's `VortexWriter.finish()` reports them). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` | +| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, null and NaN counts and uncompressed size, accumulated over the batches, as vortex-jni's `VortexWriter.finish()` reports them; a UTF-8 or binary min/max is a 64-byte bound, never exact). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` | | `close()` | `finish()` unless it already ran | Batches are not chunks: as Rust's writer does, each primitive, boolean, decimal, string and binary @@ -236,7 +236,7 @@ returned. The filter's columns are decoded even when they are not projected. An integer column takes an integer operand of any width (`Long`, `Integer`, `Short`, `Byte`). A `Double`, `Float` or `BigDecimal` operand on an integer column fails the scan with -`VortexException`, as vortex-jni refuses it ("Cannot compare different DTypes i64 and f64"). +`VortexException`, as vortex-jni refuses it ("Comparison requires equal dtypes ignoring nullability, got i64 and f64"). | Static factory | Builds | |------------------------------|------------------------------------| diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java index 5972ca648..6c6444880 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java @@ -2,13 +2,12 @@ import java.util.List; -/// Trains an FSST [Compressor] from a corpus of rows as Rust's `fsst-rs` does (`Compressor::train`, +/// Trains an FSST [Compressor] from a corpus of rows as Rust's `fsst-rs` 0.7 does (`Compressor::train`, /// the FSST paper's Algorithm 3 with the C++ reference's refinements): the same sample, the same five /// generations over a growing share of its lines, and the same candidates, gains and cut. /// -/// Training is deterministic: the same rows always produce a byte-identical [Compressor]. Rust's own -/// tables differ from this one only in where candidates of exactly equal gain and length are -/// ordered, which depends on its hash map ([TrainingGeneration]). +/// Training is deterministic: the same rows always produce a byte-identical [Compressor], the table +/// fsst-rs 0.7 trains from them ([TrainingGeneration]). /// /// ```java /// Compressor compressor = new CompressorBuilder().train(rows); diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java index a4f6608db..6ab001304 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java @@ -1,7 +1,9 @@ package io.github.dfa1.vortex.fsst; import java.util.ArrayList; +import java.util.Arrays; import java.util.BitSet; +import java.util.Comparator; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -18,11 +20,9 @@ /// /// Make-table turns every counted code into a candidate, the symbol itself and, until the last /// generation, its merges with the codes seen after it, and sums the gains of candidates that are the -/// same symbol. The candidates are popped best first from a heap that orders them by gain and length -/// only, as Rust's `BinaryHeap` does, and the first 255 that fit become the table. Two candidates of -/// equal gain and length leave the heap in an order that depends on the order they were pushed, which -/// in Rust is its hash map's iteration order; here it is the order the codes were counted. A table -/// then differs from Rust's in at most such ties. +/// same symbol. The candidates are taken best first, by gain, then length, then the symbol's bytes +/// (fsst-rs 0.7, whose ordering no longer depends on its hash map), and the first 255 that fit become +/// the table. final class TrainingGeneration { /// Maximum number of real symbols kept per generation (codes `0..254`; `0xFF` is the escape). @@ -141,18 +141,19 @@ private static List optimize(Compressor current, Counts counts, int frac } } - CandidateHeap heap = new CandidateHeap(gains.size()); - for (Map.Entry candidate : gains.entrySet()) { - heap.push(candidate.getValue(), lengthOf(candidate.getKey()), candidate.getKey()); + Candidate[] candidates = new Candidate[gains.size()]; + int index = 0; + for (Map.Entry entry : gains.entrySet()) { + candidates[index++] = new Candidate(entry.getValue(), lengthOf(entry.getKey()), entry.getKey()); } + Arrays.sort(candidates, Candidate.BEST_FIRST); BitSet takenSlots = new BitSet(); List result = new ArrayList<>(MAX_SYMBOLS); - while (!heap.isEmpty() && result.size() < MAX_SYMBOLS) { - heap.pop(); - long symbol = heap.poppedSymbol(); - int length = heap.poppedLength(); + for (int i = 0; i < candidates.length && result.size() < MAX_SYMBOLS; i++) { + long symbol = candidates[i].symbol(); + int length = candidates[i].length(); if (prune) { - long saves = heap.poppedGain(); + long saves = candidates[i].gain(); if (length == 1) { saves /= SINGLE_BYTE_GAIN_BOOST; } @@ -192,128 +193,20 @@ private static boolean claimSlot(BitSet takenSlots, long packed) { return true; } - /// Rust's `BinaryHeap`, sift for sift, ordered by `(gain, length)`: the order equal - /// candidates pop in is the heap's own, so it has to be the same algorithm. - static final class CandidateHeap { - - private final long[] gains; - private final int[] lengths; - private final long[] symbols; - private int size; - private long poppedGain; - private int poppedLength; - private long poppedSymbol; - - CandidateHeap(int capacity) { - gains = new long[capacity]; - lengths = new int[capacity]; - symbols = new long[capacity]; - } - - boolean isEmpty() { - return size == 0; - } - - long poppedGain() { - return poppedGain; - } - - int poppedLength() { - return poppedLength; - } - - long poppedSymbol() { - return poppedSymbol; - } - - void push(long gain, int length, long symbol) { - gains[size] = gain; - lengths[size] = length; - symbols[size] = symbol; - siftUp(0, size); - size++; - } - - /// Rust's `pop`: takes the last element, swaps it with the root, and sifts the root down to - /// the bottom and back up. - void pop() { - size--; - long gain = gains[size]; - int length = lengths[size]; - long symbol = symbols[size]; - if (size > 0) { - long rootGain = gains[0]; - int rootLength = lengths[0]; - long rootSymbol = symbols[0]; - gains[0] = gain; - lengths[0] = length; - symbols[0] = symbol; - gain = rootGain; - length = rootLength; - symbol = rootSymbol; - siftDownToBottom(0); - } - poppedGain = gain; - poppedLength = length; - poppedSymbol = symbol; - } - - private boolean lessOrEqual(long gainA, int lengthA, long gainB, int lengthB) { - if (gainA != gainB) { - return gainA < gainB; - } - return lengthA <= lengthB; - } - - private void siftUp(int start, int position) { - long gain = gains[position]; - int length = lengths[position]; - long symbol = symbols[position]; - int pos = position; - while (pos > start) { - int parent = (pos - 1) / 2; - if (lessOrEqual(gain, length, gains[parent], lengths[parent])) { - break; - } - gains[pos] = gains[parent]; - lengths[pos] = lengths[parent]; - symbols[pos] = symbols[parent]; - pos = parent; - } - gains[pos] = gain; - lengths[pos] = length; - symbols[pos] = symbol; - } - - private void siftDownToBottom(int position) { - int end = size; - int start = position; - long gain = gains[position]; - int length = lengths[position]; - long symbol = symbols[position]; - int pos = position; - int child = 2 * pos + 1; - while (child <= Math.max(0, end - 2)) { - if (lessOrEqual(gains[child], lengths[child], gains[child + 1], lengths[child + 1])) { - child++; - } - gains[pos] = gains[child]; - lengths[pos] = lengths[child]; - symbols[pos] = symbols[child]; - pos = child; - child = 2 * pos + 1; - } - if (child == end - 1) { - gains[pos] = gains[child]; - lengths[pos] = lengths[child]; - symbols[pos] = symbols[child]; - pos = child; - } - gains[pos] = gain; - lengths[pos] = length; - symbols[pos] = symbol; - siftUp(start, pos); - } + /// Rust's `Candidate`: a symbol and the bytes it would save. Candidates are taken best first by + /// gain, then length, then the symbol's packed bytes as an unsigned `u64` (fsst-rs 0.7's + /// `Candidate::cmp`). The symbols are distinct, so no two candidates tie and the order is total. + /// + /// @param gain the bytes the symbol saves over the sample + /// @param length the symbol's length in bytes + /// @param symbol the symbol's bytes, packed LSB-first + record Candidate(long gain, int length, long symbol) { + + /// Highest gain first, then the longer symbol, then the larger packed bytes. + static final Comparator BEST_FIRST = Comparator.comparingLong(Candidate::gain) + .thenComparingInt(Candidate::length) + .thenComparing(Candidate::symbol, Long::compareUnsigned) + .reversed(); } /// Match counts and adjacent-pair counts over the extended code space, flat arrays with no diff --git a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateHeapTest.java b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateHeapTest.java deleted file mode 100644 index 9bb5cde12..000000000 --- a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateHeapTest.java +++ /dev/null @@ -1,90 +0,0 @@ -package io.github.dfa1.vortex.fsst; - -import org.junit.jupiter.api.Test; - -import java.util.ArrayList; -import java.util.List; -import java.util.Random; - -import static org.assertj.core.api.Assertions.assertThat; - -/// The heap equal candidates pop out of, as Rust's `BinaryHeap` pops them. -class CandidateHeapTest { - - @Test - void pop_candidatesOfEqualGain_leaveInRustsHeapOrderNotPushOrder() { - // Given A5, B3, C5, D9 pushed in that order. By Rust's push (sift_up stops at an equal - // parent) and pop (swap in the last, sift down to the bottom, sift back up): D, C, A, B - TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(4); - sut.push(5, 1, 'A'); - sut.push(3, 1, 'B'); - sut.push(5, 1, 'C'); - sut.push(9, 1, 'D'); - - // When - List popped = drain(sut); - - // Then - assertThat(popped).containsExactly((long) 'D', (long) 'C', (long) 'A', (long) 'B'); - } - - @Test - void pop_allEqual_leavesInPushOrder() { - // Given three candidates of the same gain and length - TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(3); - sut.push(1, 1, 'A'); - sut.push(1, 1, 'B'); - sut.push(1, 1, 'C'); - - // When - List popped = drain(sut); - - // Then - assertThat(popped).containsExactly((long) 'A', (long) 'B', (long) 'C'); - } - - @Test - void pop_orderedByGainThenLength() { - // Given a longer symbol and a shorter one of the same gain - TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(3); - sut.push(7, 2, 'S'); - sut.push(7, 5, 'L'); - sut.push(9, 1, 'G'); - - // When - List popped = drain(sut); - - // Then the higher gain first, then the longer symbol - assertThat(popped).containsExactly((long) 'G', (long) 'L', (long) 'S'); - } - - @Test - void pop_randomCandidates_yieldsEachOnceInNonIncreasingOrder() { - // Given 500 candidates with many equal gains - Random random = new Random(11); - TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(500); - long[] gains = new long[500]; - for (int i = 0; i < gains.length; i++) { - gains[i] = random.nextInt(40); - sut.push(gains[i], 1 + random.nextInt(8), i); - } - - // When - List popped = drain(sut); - - // Then - assertThat(popped).hasSize(500).doesNotHaveDuplicates(); - for (int i = 1; i < popped.size(); i++) { - assertThat(gains[(int) (long) popped.get(i)]).isLessThanOrEqualTo(gains[(int) (long) popped.get(i - 1)]); - } - } - - private static List drain(TrainingGeneration.CandidateHeap heap) { - List popped = new ArrayList<>(); - while (!heap.isEmpty()) { - heap.pop(); - popped.add(heap.poppedSymbol()); - } - return popped; - } -} diff --git a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateTest.java b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateTest.java new file mode 100644 index 000000000..bc0e41562 --- /dev/null +++ b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateTest.java @@ -0,0 +1,95 @@ +package io.github.dfa1.vortex.fsst; + +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Random; + +import static org.assertj.core.api.Assertions.assertThat; + +/// The order training takes candidates in, as fsst-rs 0.7's `Candidate::cmp` orders them. +class CandidateTest { + + @Test + void bestFirst_higherGainFirst() { + // Given + TrainingGeneration.Candidate low = new TrainingGeneration.Candidate(3, 1, 'A'); + TrainingGeneration.Candidate high = new TrainingGeneration.Candidate(9, 1, 'B'); + + // When + List sorted = sorted(low, high); + + // Then + assertThat(sorted).containsExactly(high, low); + } + + @Test + void bestFirst_equalGain_longerFirst() { + // Given + TrainingGeneration.Candidate shorter = new TrainingGeneration.Candidate(7, 2, 0x4141); + TrainingGeneration.Candidate longer = new TrainingGeneration.Candidate(7, 5, 0x4141414141L); + + // When + List sorted = sorted(shorter, longer); + + // Then + assertThat(sorted).containsExactly(longer, shorter); + } + + @Test + void bestFirst_equalGainAndLength_largerSymbolFirst() { + // Given "aa" < "bb" as little-endian u64, so "bb" goes first + TrainingGeneration.Candidate aa = new TrainingGeneration.Candidate(10, 2, 0x6161); + TrainingGeneration.Candidate bb = new TrainingGeneration.Candidate(10, 2, 0x6262); + + // When + List sorted = sorted(aa, bb); + + // Then + assertThat(sorted).containsExactly(bb, aa); + } + + @Test + void bestFirst_comparesSymbolsUnsigned() { + // Given an 8-byte symbol whose last byte has the sign bit set, which is the larger u64 + TrainingGeneration.Candidate high = new TrainingGeneration.Candidate(10, 8, 0xFF41414141414141L); + TrainingGeneration.Candidate low = new TrainingGeneration.Candidate(10, 8, 0x4141414141414141L); + + // When + List sorted = sorted(low, high); + + // Then + assertThat(sorted).containsExactly(high, low); + } + + @Test + void bestFirst_anyInsertionOrder_sameSequence() { + // Given 200 distinct candidates with many equal gains and lengths, in random orders + Random random = new Random(11); + List candidates = new ArrayList<>(); + for (int i = 0; i < 200; i++) { + candidates.add(new TrainingGeneration.Candidate(random.nextInt(4), 1 + random.nextInt(2), i)); + } + List expected = new ArrayList<>(candidates); + expected.sort(TrainingGeneration.Candidate.BEST_FIRST); + + for (int round = 0; round < 10; round++) { + Collections.shuffle(candidates, random); + + // When + List actual = new ArrayList<>(candidates); + actual.sort(TrainingGeneration.Candidate.BEST_FIRST); + + // Then + assertThat(actual).isEqualTo(expected); + } + } + + private static List sorted(TrainingGeneration.Candidate... candidates) { + List out = new ArrayList<>(List.of(candidates)); + out.sort(TrainingGeneration.Candidate.BEST_FIRST); + return out; + } +} diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/ColumnNameEdgeCasesIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/ColumnNameEdgeCasesIntegrationTest.java index 2cc8d60ff..b66c92b98 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/ColumnNameEdgeCasesIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/ColumnNameEdgeCasesIntegrationTest.java @@ -80,7 +80,7 @@ void jniWriterRejectsDuplicateColumnNames(@TempDir Path tmp) { // Then — the reference writer refuses to produce a duplicate-name file assertThat(result) .isInstanceOf(IOException.class) - .hasRootCauseMessage("Vortex Error: Other error: StructLayout must have unique field names"); + .hasRootCauseMessage("Vortex Error: StructLayout must have unique field names"); } // ── helpers ─────────────────────────────────────────────────────────────── diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java index 5ea7eabf7..d0ee8d4c1 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java @@ -20,6 +20,8 @@ import io.github.dfa1.vortex.core.model.TimeUnitMetadata; import io.github.dfa1.vortex.writer.VortexWriter; import io.github.dfa1.vortex.writer.WriteOptions; +import io.github.dfa1.vortex.writer.encode.FixedSizeListData; +import io.github.dfa1.vortex.writer.encode.ListData; import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder; import io.github.dfa1.vortex.writer.encode.NullableData; import io.github.dfa1.vortex.writer.encode.StructData; @@ -40,6 +42,8 @@ import org.apache.arrow.vector.VarBinaryVector; import org.apache.arrow.vector.VarCharVector; import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.complex.FixedSizeListVector; +import org.apache.arrow.vector.complex.ListVector; import org.apache.arrow.vector.complex.StructVector; import org.apache.arrow.vector.ipc.ArrowReader; import org.apache.arrow.vector.types.DateUnit; @@ -70,13 +74,12 @@ import static org.assertj.core.api.Assertions.assertThat; /// The file statistics in the postscript's `statistics` segment, against what vortex-jni writes for the -/// same batches (#664): one `ArrayStats` per top-level field holding Rust's `PRUNING_STATS` (min, max, -/// sum, null count, NaN count), each taken per batch and folded. The data hits every rule of that -/// fold: a string min cut to 64 bytes in one batch only (inexact min, exact max), a string column -/// with an all-null batch and a binary max that cannot be rounded up (both lose their max), an `i64` -/// sum that overflows (no sum), decimals of two widths (the sum is wider than the values), an -/// all-null float column, an extension (storage min/max, no sum) and a nullable struct (null count -/// only). +/// same batches (#664): one `ArrayStats` per top-level field holding Rust 0.91's default aggregates +/// (min, max, null count, NaN count, uncompressed size), each accumulated batch by batch. The data +/// hits every rule: a string min cut to 64 bytes in one batch only, a string column with an +/// all-null batch (it keeps its max), a binary max that cannot be rounded up (no max), string and +/// binary bounds always inexact, decimals of two widths, an all-null float column (a constant +/// validity), an extension (storage min/max) and a nullable struct (null count and size). class FileStatisticsInteropIntegrationTest { private static final Session SESSION = Session.create(); @@ -150,8 +153,76 @@ void nullColumn_countsEveryRowNull(@TempDir Path tmp) throws Exception { assertThat(result).isEqualTo(fileStatistics(jni)); } + @Test + void noBatch_writesWhatJniWrites(@TempDir Path tmp) throws Exception { + // Given — a file without any batch + Schema arrow = new Schema(List.of(Field.nullable("i", new ArrowType.Int(64, true)), Field.nullable("s", ArrowType.Utf8.INSTANCE))); + Path jni = tmp.resolve("jni.vortex"); + dev.vortex.api.VortexWriter.builder(SESSION, jni.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR) + .build().close(); + Path java = tmp.resolve("java.vortex"); + List names = List.of(ColumnName.of("i"), ColumnName.of("s")); + DType.Struct schema = new DType.Struct(names, List.of(DTypes.I64.asNullable(), DTypes.UTF8.asNullable()), false); + try (FileChannel ch = FileChannel.open(java, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { + VortexWriter.create(ch, schema, WriteOptions.defaults()).close(); + } + + // When + List result = fileStatistics(java, names); + + // Then + assertThat(result).isEqualTo(fileStatistics(jni, names)); + } + + @Test + void listColumns_sizeTheElementsTheirValidListsUse(@TempDir Path tmp) throws Exception { + // Given — a nullable list of nullable i32 with null lists and null elements, a list of + // strings some too long to inline, and a nullable fixed-size list: the uncompressed size + // counts the elements of valid lists, 16 bytes of offset and size per row, and validities + List names = List.of(ColumnName.of("l"), ColumnName.of("ls"), ColumnName.of("fsl")); + Path jni = tmp.resolve("jni.vortex"); + writeListsJni(jni); + Path java = tmp.resolve("java.vortex"); + DType.Struct schema = new DType.Struct(names, List.of( + new DType.List(DTypes.I32.asNullable(), true), + new DType.List(DTypes.UTF8, false), + new DType.FixedSizeList(DTypes.I32, 2, true)), false); + try (FileChannel ch = FileChannel.open(java, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + VortexWriter w = VortexWriter.create(ch, schema, WriteOptions.defaults())) { + for (int batch = 0; batch < BATCHES; batch++) { + w.writeChunk(javaListBatch(names, batch)); + } + } + + // When + List result = fileStatistics(java, names); + + // Then + assertThat(result).isEqualTo(fileStatistics(jni, names)); + } + // ── data ──────────────────────────────────────────────────────────────── + private static boolean listValid(int row) { + return row % 4 != 0; + } + + private static int listLength(int row) { + return row % 3; + } + + private static boolean elementValid(int row, int k) { + return (row + k) % 5 != 0; + } + + private static String listString(int row, int k) { + return k == 1 ? "long-enough-" + row : "s" + k; + } + + private static boolean fslValid(int row) { + return row % 6 != 0; + } + private static boolean i32Valid(int row) { return row % 7 != 0; } @@ -346,11 +417,121 @@ private static Map javaBatch(int batch) { return chunk; } + private static void writeListsJni(Path file) throws Exception { + Schema arrow = new Schema(List.of( + new Field("l", FieldType.nullable(ArrowType.List.INSTANCE), + List.of(Field.nullable("item", new ArrowType.Int(32, true)))), + new Field("ls", FieldType.notNullable(ArrowType.List.INSTANCE), + List.of(Field.notNullable("item", ArrowType.Utf8.INSTANCE))), + new Field("fsl", FieldType.nullable(new ArrowType.FixedSizeList(2)), + List.of(Field.notNullable("item", new ArrowType.Int(32, true)))))); + try (dev.vortex.api.VortexWriter w = dev.vortex.api.VortexWriter.builder(SESSION, + file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) { + for (int batch = 0; batch < BATCHES; batch++) { + try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) { + ListVector l = (ListVector) root.getVector("l"); + ListVector ls = (ListVector) root.getVector("ls"); + FixedSizeListVector fsl = (FixedSizeListVector) root.getVector("fsl"); + l.allocateNew(); + ls.allocateNew(); + fsl.allocateNew(); + IntVector lItems = (IntVector) l.getDataVector(); + VarCharVector lsItems = (VarCharVector) ls.getDataVector(); + IntVector fslItems = (IntVector) fsl.getDataVector(); + int lCount = 0; + int lsCount = 0; + for (int i = 0; i < ROWS; i++) { + int row = batch * ROWS + i; + if (listValid(row)) { + int start = l.startNewValue(i); + for (int k = 0; k < listLength(row); k++) { + if (elementValid(row, k)) { + lItems.setSafe(start + k, row + k); + } else { + lItems.setNull(start + k); + } + } + lCount = start + listLength(row); + l.endValue(i, listLength(row)); + } else { + l.setNull(i); + } + int start = ls.startNewValue(i); + for (int k = 0; k < listLength(row); k++) { + lsItems.setSafe(start + k, listString(row, k).getBytes(StandardCharsets.UTF_8)); + } + lsCount = start + listLength(row); + ls.endValue(i, listLength(row)); + fslItems.setSafe(i * 2, row); + fslItems.setSafe(i * 2 + 1, -row); + if (fslValid(row)) { + fsl.setNotNull(i); + } else { + fsl.setNull(i); + } + } + lItems.setValueCount(lCount); + lsItems.setValueCount(lsCount); + fslItems.setValueCount(ROWS * 2); + root.setRowCount(ROWS); + try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); + ArrowSchema exported = ArrowSchema.allocateNew(ALLOCATOR)) { + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, exported); + w.writeBatch(array.memoryAddress(), exported.memoryAddress()); + } + } + } + } + } + + private static Map javaListBatch(List names, int batch) { + long[] lOffsets = new long[ROWS + 1]; + long[] lsOffsets = new long[ROWS + 1]; + boolean[] lValid = new boolean[ROWS]; + boolean[] fslValidity = new boolean[ROWS]; + List lItems = new ArrayList<>(); + List lsItems = new ArrayList<>(); + int[] fslItems = new int[ROWS * 2]; + for (int i = 0; i < ROWS; i++) { + int row = batch * ROWS + i; + lValid[i] = listValid(row); + int lLength = lValid[i] ? listLength(row) : 0; + for (int k = 0; k < lLength; k++) { + lItems.add(elementValid(row, k) ? row + k : null); + } + lOffsets[i + 1] = lOffsets[i] + lLength; + for (int k = 0; k < listLength(row); k++) { + lsItems.add(listString(row, k)); + } + lsOffsets[i + 1] = lsOffsets[i] + listLength(row); + fslItems[i * 2] = row; + fslItems[i * 2 + 1] = -row; + fslValidity[i] = fslValid(row); + } + int[] lValues = new int[lItems.size()]; + boolean[] lElementValid = new boolean[lItems.size()]; + for (int k = 0; k < lValues.length; k++) { + lElementValid[k] = lItems.get(k) != null; + lValues[k] = lElementValid[k] ? lItems.get(k) : 0; + } + Map chunk = new LinkedHashMap<>(); + chunk.put(names.get(0), new NullableData( + new ListData(new NullableData(lValues, lElementValid), lOffsets, ROWS), lValid)); + chunk.put(names.get(1), new ListData(lsItems.toArray(String[]::new), lsOffsets, ROWS)); + chunk.put(names.get(2), new NullableData(new FixedSizeListData(fslItems, ROWS), fslValidity)); + return chunk; + } + // ── readers ───────────────────────────────────────────────────────────── /// Each field's `ArrayStats` table as written: every stat's presence, the raw scalar bytes and /// each bound's precision. private static List fileStatistics(Path file) throws IOException { + return fileStatistics(file, NAMES); + } + + /// Each field's `ArrayStats` table as written, named after `names`. + private static List fileStatistics(Path file, List names) throws IOException { MemorySegment seg = MemorySegment.ofArray(Files.readAllBytes(file)); long size = seg.byteSize(); int psLen = Short.toUnsignedInt(seg.get(VortexFormat.LE_SHORT, size - VortexFormat.TRAILER_SIZE + 2)); @@ -361,7 +542,7 @@ private static List fileStatistics(Path file) throws IOException { List fields = new ArrayList<>(); for (int j = 0; j < fs.fieldStatsLength(); j++) { FbsArrayStats a = fs.fieldStats(j); - fields.add(NAMES.get(j) + ": min=" + hex(a.minAsSegment()) + "/" + a.minPrecision() + fields.add(names.get(j) + ": min=" + hex(a.minAsSegment()) + "/" + a.minPrecision() + " max=" + hex(a.maxAsSegment()) + "/" + a.maxPrecision() + " sum=" + hex(a.sumAsSegment()) + " nulls=" + (a.hasNullCount() ? a.nullCount() : "-") diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/FractionalOperandInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/FractionalOperandInteropIntegrationTest.java index 06945dced..0782632ac 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/FractionalOperandInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/FractionalOperandInteropIntegrationTest.java @@ -56,7 +56,7 @@ void fractionalOperandOnIntegerColumn_bothReadersReject(@TempDir Path tmp) throw // When / Then assertThatThrownBy(() -> scanJni(file)) .as("vortex-jni") - .hasMessageContaining("Cannot compare different DTypes i64 and f64"); + .hasMessageContaining("Comparison requires equal dtypes ignoring nullability, got i64 and f64"); assertThatThrownBy(() -> scanJava(file)) .as("vortex-java") .isInstanceOf(VortexException.class) diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java index ad27f79d4..56c06610c 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java @@ -29,8 +29,6 @@ import java.nio.charset.StandardCharsets; import java.nio.file.Path; import java.nio.file.StandardOpenOption; -import java.util.ArrayList; -import java.util.Collections; import java.util.List; import java.util.Map; import java.util.function.IntFunction; @@ -38,12 +36,9 @@ import static org.assertj.core.api.Assertions.assertThat; -/// vortex-java trains the FSST symbol table as Rust's `fsst-rs` does (#583): the same sample, the same -/// five generations, the same candidates, gains and cut. The two tables are not byte for byte equal, -/// and cannot be: candidates of exactly equal gain leave Rust's heap in the order its hash map -/// iterates them, which depends on the CPU's SIMD width, so Rust's own table differs between x86 and -/// arm. What has to hold is that nearly every symbol is Rust's and the code stream is as long; -/// [NullableTextInteropIntegrationTest] holds a column whose whole encoding tree equals Rust's. +/// vortex-java trains the FSST symbol table as Rust's `fsst-rs` 0.7 does (#583): the same sample, the +/// same five generations, the same candidates, gains and cut, and the same order between candidates of +/// equal gain and length. The symbols, their lengths and the code stream are byte for byte Rust's. class FsstTrainingParityIntegrationTest { private static final Session SESSION = Session.create(); @@ -65,7 +60,7 @@ static Stream shapes() { @ParameterizedTest(name = "{0}") @MethodSource("shapes") - void javaTrainsAnFsstTableCloseToJnis(String name, int rows, IntFunction values, @TempDir Path tmp) + void javaTrainsJnisFsstTable(String name, int rows, IntFunction values, @TempDir Path tmp) throws Exception { // Given Path jniFile = tmp.resolve("jni.vortex"); @@ -80,29 +75,9 @@ void javaTrainsAnFsstTableCloseToJnis(String name, int rows, IntFunction // Then List expected = EncodingTrees.firstFlatBuffers(jniFile); List actual = EncodingTrees.firstFlatBuffers(javaFile); - List rustSymbols = symbolSet(expected); - List javaSymbols = symbolSet(actual); - long common = javaSymbols.stream().filter(rustSymbols::contains).count(); - assertThat(common).as("symbols Rust also has, of %d (Java has %d)", rustSymbols.size(), javaSymbols.size()) - .isGreaterThanOrEqualTo((long) (0.8 * rustSymbols.size())); - assertThat((double) actual.get(2).length).as("code stream bytes, Rust's is %d", expected.get(2).length) - .isBetween(0.985 * expected.get(2).length, 1.015 * expected.get(2).length); - } - - /// The table's symbols as (length, bytes) strings, sorted: the table whatever order it is numbered in. - private static List symbolSet(List buffers) { - byte[] symbols = buffers.get(0); - byte[] lengths = buffers.get(1); - List out = new ArrayList<>(); - for (int i = 0; i < lengths.length; i++) { - StringBuilder hex = new StringBuilder(); - for (int k = 0; k < lengths[i]; k++) { - hex.append(String.format("%02x", symbols[i * 8 + k])); - } - out.add(hex.toString()); - } - Collections.sort(out); - return out; + assertThat(actual.get(0)).as("symbols").isEqualTo(expected.get(0)); + assertThat(actual.get(1)).as("symbol lengths").isEqualTo(expected.get(1)); + assertThat(actual.get(2)).as("code stream").isEqualTo(expected.get(2)); } private static long mix(long i) { diff --git a/pom.xml b/pom.xml index 68711ada5..6e01dd053 100644 --- a/pom.xml +++ b/pom.xml @@ -79,7 +79,7 @@ 0.30.0 3.27.7 5.24.0 - 0.90.0 + 0.91.0 19.0.0 2.0.20 diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java b/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java index a21d8fdeb..70cf8dc62 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java @@ -14,63 +14,51 @@ import java.math.BigInteger; import java.nio.charset.StandardCharsets; import java.util.Arrays; -import java.util.OptionalLong; /// One top-level column's file statistics, written into the postscript's `statistics` segment: -/// Rust 0.90.0's `FileStatsAccumulator` (`vortex-layout/src/layouts/file_stats.rs`) over the default -/// `PRUNING_STATS` (min, max, sum, null count, NaN count), as vortex-jni writes them. +/// Rust 0.91.0's `FileStatsAccumulator` (`vortex-layout/src/layouts/file_stats.rs`) over its default +/// aggregates (max, min, null count, NaN count, uncompressed size), as vortex-jni writes them. /// -/// Rust takes every stat per `writeChunk` batch and folds the batches at the end, so this does too: -/// - a UTF-8 or binary bound is cut to 64 bytes per batch, and it is inexact when any batch's bound -/// was cut, even a batch that does not hold the extreme; -/// - a UTF-8 or binary max is dropped when any batch has none (all null, or a bound that cannot be -/// rounded up): skipping that batch would be unsound; -/// - a sum is dropped when any batch's sum, or the fold, overflows: integers in `i64`/`u64`, a -/// decimal of precision `p` past `p + 10` digits in a batch and `p + 20` in the fold (Rust sums -/// the batch sums with a second `Sum`). +/// Rust accumulates every aggregate batch by batch, so this does too: +/// - a UTF-8 or binary bound is Rust's `BoundedMin`/`BoundedMax`, cut to 64 bytes per batch, and +/// always inexact: it only approximates the `Min`/`Max` the statistics slot asks for; +/// - an all-null batch adds no bound, and a batch whose max cannot be rounded up within 64 bytes +/// leaves the column without a max; +/// - the uncompressed size is the sum of each batch's ([UncompressedSize]). /// -/// An extension column has its storage's min and max but no sum or NaN count, a struct, list or -/// fixed-size list only its null count, and a variant column no statistic at all. +/// An extension column has its storage's min and max but no NaN count, a struct, list or +/// fixed-size list only its null count and size, and a variant column no statistic at all. final class FileStatsAccumulator { - /// Rust's `MAX_PRECISION` for decimals. - private static final int MAX_DECIMAL_PRECISION = 76; - + private final DType dtype; private final Kind kind; private final PType ptype; private final DType.Decimal decimal; private final boolean extension; private final boolean nullDtype; private final boolean variant; - private boolean seenBatch; private Object min; - private boolean minTruncated; private Object max; - private boolean maxTruncated; private boolean maxDropped; - private Object sum; - private boolean sumOverflowed; private long nullCount; private long nanCount; + private long uncompressedSize; FileStatsAccumulator(DType dtype) { DType storage = dtype instanceof DType.Extension ext ? ext.storageDType() : dtype; + this.dtype = dtype; this.kind = ZoneAccumulator.kindOf(storage); this.ptype = storage instanceof DType.Primitive p ? p.ptype() : null; this.decimal = storage instanceof DType.Decimal d ? d : null; this.extension = dtype instanceof DType.Extension; this.nullDtype = storage instanceof DType.Null; this.variant = storage instanceof DType.Variant; - this.seenBatch = false; this.min = null; - this.minTruncated = false; this.max = null; - this.maxTruncated = false; this.maxDropped = false; - this.sum = null; - this.sumOverflowed = false; this.nullCount = 0; this.nanCount = 0; + this.uncompressedSize = 0; } /// Adds one `writeChunk` batch of the column; an empty batch is skipped, as Rust filters it out. @@ -81,12 +69,12 @@ void add(Object batch, long rows) { if (rows == 0 || variant) { return; } - seenBatch = true; + int n = (int) rows; + uncompressedSize += UncompressedSize.of(dtype, batch, n); if (nullDtype) { nullCount += rows; return; } - int n = (int) rows; Object data = kind == Kind.NULL_COUNT_ONLY ? ZoneAccumulator.validityOnly(batch, n) : ZoneAccumulator.comparable(batch); if (ptype != null && !(data instanceof NullableData)) { // the common case, on the writer's critical path: the vectorized kernel, no widened copy @@ -101,49 +89,32 @@ void add(Object batch, long rows) { addExtremes(extremes.min(), extremes.max()); } } - if (summable() && !sumOverflowed) { - Object values = data; - boolean[] validity = null; - if (data instanceof NullableData nd) { - values = nd.values(); - validity = nd.validity(); - } - addSum(values, validity); - } } - /// The folded stats, every one exact unless [#minExact()] or [#maxExact()] says otherwise. + /// The folded stats, every one exact unless [#boundsExact()] says otherwise. /// - /// @return the column's stats; [NodeStats#NONE] for a variant column or one without rows, whose - /// per-batch stats Rust drops as all null + /// @return the column's stats, a zero null count and size when it has no row; [NodeStats#NONE] + /// for a variant column, which Rust gives no statistic NodeStats stats() { - if (!seenBatch) { + if (variant) { return NodeStats.NONE; } - ProtoScalarValue sumScalar = null; - if (summable() && !sumOverflowed) { - sumScalar = sumScalar(sum); - } - return new NodeStats(scalar(min), scalar(maxValue()), null, null, null, nullCount, null, nans(), sumScalar); + return new NodeStats(scalar(min), scalar(maxValue()), null, null, null, nullCount, uncompressedSize, nans()); } - /// The folded stats as Java values: a `Long` for an integer (a `U64` as its bits) or a bool sum, - /// a `Float` for an `F16`/`F32` bound and a `Double` for an `F64` one or any float sum, a - /// `Boolean`, a `BigDecimal`, a `String` for UTF-8 and a `byte[]` for binary. + /// The folded stats as Java values: a `Long` for an integer (a `U64` as its bits), a `Float` for + /// an `F16`/`F32` bound and a `Double` for an `F64` one, a `Boolean`, a `BigDecimal`, a `String` + /// for UTF-8 and a `byte[]` for binary. /// /// @return the same stats as [#stats()]; [ArrayStats#empty()] where that is [NodeStats#NONE] ArrayStats arrayStats() { - if (!seenBatch) { + if (variant) { return ArrayStats.empty(); } - Object sumValue = null; - if (summable() && !sumOverflowed) { - sumValue = sum; - } Object lo = javaValue(min); Object hi = javaValue(maxValue()); - return new ArrayStats(lo, lo != null && !minTruncated, hi, hi != null && !maxTruncated, sumValue, - null, null, null, nullCount, null, nans()); + return new ArrayStats(lo, lo != null && boundsExact(), hi, hi != null && boundsExact(), null, + null, null, null, nullCount, uncompressedSize, nans()); } private Object maxValue() { @@ -177,18 +148,12 @@ private Object javaValue(Object value) { }; } - /// Whether the min is the column's exact minimum: no batch cut its bound. + /// Whether the min and max are the column's exact extremes: they are, but for a UTF-8 or binary + /// column, whose `BoundedMin`/`BoundedMax` Rust writes as inexact even when no bound was cut. /// - /// @return `false` when a UTF-8 or binary batch minimum was longer than 64 bytes - boolean minExact() { - return !minTruncated; - } - - /// Whether the max is the column's exact maximum: no batch cut its bound. - /// - /// @return `false` when a UTF-8 or binary batch maximum was longer than 64 bytes - boolean maxExact() { - return !maxTruncated; + /// @return `false` for a UTF-8 or binary column + boolean boundsExact() { + return kind != Kind.UTF8 && kind != Kind.BINARY; } private void addExtremes(Object lo, Object hi) { @@ -256,116 +221,25 @@ private static long nanCount(Object values) { private void addBounds(byte[] lo, byte[] hi) { if (lo == null) { - // an all-null batch: its null max makes the column max unsound, its null min is skipped - maxDropped = true; + // an all-null batch has no bound to add return; } int limit = ZoneAccumulator.BOUNDED_MAX_BYTES; byte[] lower = kind == Kind.UTF8 ? ZoneAccumulator.utf8LowerBound(lo) : Arrays.copyOf(lo, Math.min(lo.length, limit)); byte[] upper = kind == Kind.UTF8 ? ZoneAccumulator.utf8UpperBound(hi) : ZoneAccumulator.binaryUpperBound(hi); - minTruncated |= lo.length > limit; if (min == null || Arrays.compareUnsigned(lower, (byte[]) min) < 0) { min = lower; } if (upper == null) { + // Rust's BoundedMax state `Unknown`: no upper bound, now or after any later batch maxDropped = true; return; } - maxTruncated |= hi.length > limit; if (max == null || Arrays.compareUnsigned(upper, (byte[]) max) > 0) { max = upper; } } - private boolean summable() { - return !extension && (kind == Kind.INT || kind == Kind.F16 || kind == Kind.FLOAT - || kind == Kind.BOOL || kind == Kind.DECIMAL); - } - - private void addSum(Object values, boolean[] validity) { - Object batchSum = switch (kind) { - case INT -> { - OptionalLong s = SimdOperationsSupport.preferred().sum(values, ptype); - yield s.isPresent() ? s.getAsLong() : null; - } - case F16, FLOAT -> SimdOperationsSupport.preferred().sumFloating(values, ptype); - case BOOL -> trueCount((boolean[]) values, validity); - case DECIMAL -> decimalSum((BigDecimal[]) values, validity, decimal.precision() + 10); - default -> throw new IllegalStateException("no sum for " + kind); - }; - Object folded = batchSum == null ? null : fold(sum == null ? zeroSum() : sum, batchSum); - if (folded == null) { - sumOverflowed = true; - } - sum = folded; - } - - /// `acc + batch` in the sum's type, or `null` on overflow. - private Object fold(Object acc, Object batch) { - return switch (kind) { - case INT -> ptype.isUnsigned() ? addUnsigned((Long) acc, (Long) batch) : addSigned((Long) acc, (Long) batch); - case BOOL -> addUnsigned((Long) acc, (Long) batch); - case F16, FLOAT -> (Double) acc + (Double) batch; - case DECIMAL -> fitsPrecision(((BigDecimal) acc).add((BigDecimal) batch), decimal.precision() + 20); - default -> throw new IllegalStateException("no sum for " + kind); - }; - } - - private Object zeroSum() { - return switch (kind) { - case F16, FLOAT -> 0.0; - case DECIMAL -> BigDecimal.ZERO.setScale(decimal.scale()); - default -> 0L; - }; - } - - private static Long addSigned(long a, long b) { - long r = a + b; - // overflow iff both operands have the sign the result lacks - if (((a ^ r) & (b ^ r)) < 0) { - return null; - } - return r; - } - - private static Long addUnsigned(long a, long b) { - long r = a + b; - if (Long.compareUnsigned(r, a) < 0) { - return null; - } - return r; - } - - private static long trueCount(boolean[] values, boolean[] validity) { - long count = 0; - for (int i = 0; i < values.length; i++) { - if (values[i] && (validity == null || validity[i])) { - count++; - } - } - return count; - } - - private static BigDecimal decimalSum(BigDecimal[] values, boolean[] validity, int precision) { - BigDecimal total = BigDecimal.ZERO; - for (int i = 0; i < values.length; i++) { - if (values[i] != null && (validity == null || validity[i])) { - total = total.add(values[i]); - } - } - return fitsPrecision(total, precision); - } - - /// `value`, or `null` when its unscaled value has more digits than `precision` (capped at - /// Rust's maximum): Rust's `fits_in_precision` check on a decimal sum. - private static BigDecimal fitsPrecision(BigDecimal value, int precision) { - int digits = Math.min(precision, MAX_DECIMAL_PRECISION); - if (value.unscaledValue().abs().compareTo(BigInteger.TEN.pow(digits)) >= 0) { - return null; - } - return value; - } - private int compare(Object a, Object b) { return switch (kind) { case INT -> ptype.isUnsigned() ? Long.compareUnsigned((Long) a, (Long) b) : Long.compare((Long) a, (Long) b); @@ -398,22 +272,6 @@ private ProtoScalarValue scalar(Object value) { }; } - /// A sum as the scalar of Rust's sum dtype: `i64`, `u64` (unsigned and bool), `f64`, or a - /// decimal of precision `p + 20`. - private ProtoScalarValue sumScalar(Object value) { - return switch (kind) { - case INT -> ptype.isUnsigned() ? ProtoScalarValue.ofUint64Value((Long) value) : ProtoScalarValue.ofInt64Value((Long) value); - case BOOL -> ProtoScalarValue.ofUint64Value((Long) value); - case F16, FLOAT -> ProtoScalarValue.ofF64Value((Double) value); - case DECIMAL -> { - int precision = Math.min(decimal.precision() + 20, MAX_DECIMAL_PRECISION); - int width = DType.Decimal.of(precision, decimal.scale(), true).byteWidth(); - yield ProtoScalarValue.ofBytesValue(littleEndian(((BigDecimal) value).unscaledValue(), width)); - } - default -> throw new IllegalStateException("no sum for " + kind); - }; - } - private static byte[] littleEndian(BigInteger unscaled, int width) { // toByteArray is big-endian and minimal: reverse it, then sign-extend to the width byte[] be = unscaled.toByteArray(); diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/Repartitioner.java b/writer/src/main/java/io/github/dfa1/vortex/writer/Repartitioner.java index 294bffb64..6bd1d967f 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/Repartitioner.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/Repartitioner.java @@ -183,7 +183,7 @@ private Object drainPending() { /// Rust's exact `uncompressed_size_in_bytes` of the block being filled, as one canonical /// array (`vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes`): fixed-width values /// at their width, Utf8/Binary as a 16-byte view per row plus each valid value too long to - /// inline in its view, and the validity per [#validityBytes(long,long)]. + /// inline in its view, and the validity per [UncompressedSize#validityBytes(long,long)]. private long blockBytes() { long outOfLine = 0; long valid = 0; @@ -197,20 +197,7 @@ private long blockBytes() { case DType.Bool _ -> (blockRows + 7L) / 8; default -> (long) blockRows * elementBytes(storage); }; - return values + validityBytes(valid, blockRows); - } - - /// Rust's `validity_uncompressed_size_in_bytes` of a block's mask: nothing when every row is - /// valid, a bitmap when some are, and when none are, a `ConstantArray(false)`'s one buffer -- - /// the 2-byte proto `ScalarValue { bool_value: false }`. - private static long validityBytes(long valid, long rows) { - if (valid == rows) { - return 0; - } - if (valid == 0) { - return 2; - } - return (rows + 7) / 8; + return values + UncompressedSize.validityBytes(valid, blockRows); } private static long validRows(Piece p) { diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java b/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java index 169f95754..faf597eb3 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java @@ -412,7 +412,7 @@ private static ByteBuffer buildFileStatistics(List fields) int[] offsets = new int[fields.size()]; for (int i = 0; i < fields.size(); i++) { FileStatsAccumulator field = fields.get(i); - offsets[i] = buildStats(fbb, field.stats(), field.minExact(), field.maxExact()); + offsets[i] = buildStats(fbb, field.stats(), field.boundsExact(), field.boundsExact()); } int vec = FbsFileStatistics.createFieldStatsVector(fbb, offsets); fbb.finish(FbsFileStatistics.createFbsFileStatistics(fbb, vec)); diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/UncompressedSize.java b/writer/src/main/java/io/github/dfa1/vortex/writer/UncompressedSize.java new file mode 100644 index 000000000..21c55fb27 --- /dev/null +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/UncompressedSize.java @@ -0,0 +1,248 @@ +package io.github.dfa1.vortex.writer; + +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.writer.encode.FixedSizeListData; +import io.github.dfa1.vortex.writer.encode.ListData; +import io.github.dfa1.vortex.writer.encode.ListViewData; +import io.github.dfa1.vortex.writer.encode.NullableData; +import io.github.dfa1.vortex.writer.encode.StructData; + +import java.nio.charset.StandardCharsets; +import java.util.List; + +/// Rust's `uncompressed_size_in_bytes` of one batch: the bytes its rows take as a canonical array +/// (`vortex-array/src/aggregate_fn/fns/uncompressed_size_in_bytes`), the file statistic vortex-jni +/// writes per top-level field. +/// +/// - fixed-width values take their width per row (a decimal its precision's width), bools a packed bit; +/// - UTF-8 and binary take a 16-byte view per row, plus each valid value too long to inline in it; +/// - a struct is the sum of its fields, a list the elements its valid lists use plus 16 bytes of +/// offset and size per row, a fixed-size list all of its elements, an extension its storage; +/// - every array adds its validity per [#validityBytes(long, long)]. +final class UncompressedSize { + + /// Rust's `BinaryView`: 16 bytes, of which up to 12 hold the value inline. + private static final int VIEW_BYTES = 16; + private static final int MAX_INLINED_BYTES = 12; + + /// A list view's offset and size, a `u64` each per row. + private static final int LIST_VIEW_BYTES = 16; + + private UncompressedSize() { + } + + /// Whether Rust measures arrays of `dtype`: every dtype but one holding a variant or a union, + /// which no carrier of this writer holds. + /// + /// @param dtype the column dtype + /// @return `true` when [#of(DType, Object, int)] can size it + static boolean supports(DType dtype) { + return switch (dtype) { + case DType.Variant _, DType.Union _ -> false; + case DType.Extension ext -> supports(ext.storageDType()); + case DType.Struct s -> s.fieldTypes().stream().allMatch(UncompressedSize::supports); + case DType.List l -> supports(l.elementType()); + case DType.FixedSizeList f -> supports(f.elementType()); + case DType.Map m -> supports(m.keyType()) && supports(m.valueType()); + default -> true; + }; + } + + /// The size of one batch. + /// + /// @param dtype the column dtype, one [#supports(DType)] accepts + /// @param data the batch's carrier + /// @param rows the batch's row count + /// @return its uncompressed size in bytes + static long of(DType dtype, Object data, int rows) { + return size(dtype, data, Rows.range(0, rows)); + } + + /// Rust's `validity_uncompressed_size_in_bytes`: nothing when every row is valid, and when none + /// is, the one buffer of a `ConstantArray(false)` (the 2-byte proto `ScalarValue { bool_value: + /// false }`); otherwise a packed bitmap. + /// + /// @param valid the valid rows + /// @param rows all rows + /// @return the validity's size in bytes + static long validityBytes(long valid, long rows) { + if (valid == rows) { + return 0; + } + if (valid == 0) { + return 2; + } + return (rows + 7) / 8; + } + + private static long size(DType dtype, Object data, Rows rows) { + if (dtype instanceof DType.Extension ext) { + return size(ext.storageDType(), data, rows); + } + if (dtype instanceof DType.Null) { + // a null array has no buffer and no validity + return 0; + } + boolean[] validity = data instanceof NullableData nd ? nd.validity() : null; + Object values = data instanceof NullableData nd ? nd.values() : data; + long n = rows.count(); + long bytes = switch (dtype) { + case DType.Bool _ -> (n + 7) / 8; + case DType.Primitive p -> n * p.ptype().byteSize(); + case DType.Decimal d -> n * d.byteWidth(); + case DType.Utf8 _, DType.Binary _ -> VIEW_BYTES * n + outOfLineBytes(values, validity, rows); + case DType.Struct s -> fieldsBytes(s.fieldTypes(), ((StructData) values).fieldArrays(), rows); + case DType.List l -> listBytes(values, validity, rows, (elements, used) -> size(l.elementType(), elements, used)); + // a map is a list of non-nullable key-value entries + case DType.Map m -> listBytes(values, validity, rows, (entries, used) -> + fieldsBytes(List.of(m.keyType(), m.valueType()), ((StructData) entries).fieldArrays(), used)); + case DType.FixedSizeList f -> fixedSizeListBytes(f, (FixedSizeListData) values, rows); + case DType.Null _, DType.Extension _, DType.Variant _, DType.Union _ -> + throw new IllegalStateException("no uncompressed size for " + dtype); + }; + return bytes + validityBytes(validCount(validity, rows), n); + } + + private static long fieldsBytes(List types, List fields, Rows rows) { + long bytes = 0; + for (int i = 0; i < types.size(); i++) { + bytes += size(types.get(i), fields.get(i), rows); + } + return bytes; + } + + /// Rust's `list_view_uncompressed_size_in_bytes`: the elements the valid lists use, as one slice + /// when they tile one run of elements, else each counted as often as a list uses it. + private static long listBytes(Object values, boolean[] validity, Rows rows, ElementSize elementSize) { + long views = (long) LIST_VIEW_BYTES * rows.count(); + Object elements; + long[] starts = new long[rows.count()]; + long[] sizes = new long[rows.count()]; + switch (values) { + case ListData d -> { + elements = d.elements(); + for (int i = 0; i < rows.count(); i++) { + int r = rows.at(i); + starts[i] = d.offsets()[r]; + sizes[i] = d.offsets()[r + 1] - d.offsets()[r]; + } + } + case ListViewData d -> { + elements = d.elements(); + for (int i = 0; i < rows.count(); i++) { + int r = rows.at(i); + starts[i] = d.offsets()[r]; + sizes[i] = d.sizes()[r]; + } + } + default -> throw new IllegalStateException("not a list carrier: " + values.getClass().getSimpleName()); + } + long start = Long.MAX_VALUE; + long end = Long.MIN_VALUE; + long count = 0; + for (int i = 0; i < rows.count(); i++) { + if (validity == null || validity[rows.at(i)]) { + start = Math.min(start, starts[i]); + end = Math.max(end, starts[i] + sizes[i]); + count += sizes[i]; + } + } + if (start == Long.MAX_VALUE) { + // no valid list + return views; + } + Rows used; + if (count == end - start) { + used = Rows.range((int) start, (int) count); + } else { + int[] indices = new int[Math.toIntExact(count)]; + int k = 0; + for (int i = 0; i < rows.count(); i++) { + if (validity == null || validity[rows.at(i)]) { + for (long e = starts[i]; e < starts[i] + sizes[i]; e++) { + indices[k++] = (int) e; + } + } + } + used = Rows.of(indices); + } + return views + elementSize.of(elements, used); + } + + /// Rust's `fixed_size_list_uncompressed_size_in_bytes`: every element of the rows, null rows too. + private static long fixedSizeListBytes(DType.FixedSizeList dtype, FixedSizeListData data, Rows rows) { + int width = dtype.fixedSize(); + Rows elements; + if (rows.indices() == null) { + elements = Rows.range(rows.offset() * width, rows.count() * width); + } else { + int[] indices = new int[rows.count() * width]; + for (int i = 0; i < rows.count(); i++) { + for (int k = 0; k < width; k++) { + indices[i * width + k] = rows.at(i) * width + k; + } + } + elements = Rows.of(indices); + } + return size(dtype.elementType(), data.elements(), elements); + } + + /// The bytes of the valid values longer than 12, which a VarBinView keeps out of its views + /// (`BinaryView::MAX_INLINED_SIZE`). + private static long outOfLineBytes(Object values, boolean[] validity, Rows rows) { + long bytes = 0; + for (int i = 0; i < rows.count(); i++) { + int r = rows.at(i); + if (validity != null && !validity[r]) { + continue; + } + int length = switch (values) { + case String[] a -> a[r] == null ? 0 : a[r].getBytes(StandardCharsets.UTF_8).length; + case byte[][] a -> a[r] == null ? 0 : a[r].length; + default -> throw new IllegalStateException("not a string carrier: " + values.getClass().getSimpleName()); + }; + if (length > MAX_INLINED_BYTES) { + bytes += length; + } + } + return bytes; + } + + private static long validCount(boolean[] validity, Rows rows) { + if (validity == null) { + return rows.count(); + } + long valid = 0; + for (int i = 0; i < rows.count(); i++) { + if (validity[rows.at(i)]) { + valid++; + } + } + return valid; + } + + /// The size of a list's elements over the rows its valid lists use. + @FunctionalInterface + private interface ElementSize { + long of(Object elements, Rows used); + } + + /// The rows of a carrier an array holds: a run `offset..offset + count`, or the listed indices. + private record Rows(int offset, int count, int[] indices) { + + static Rows range(int offset, int count) { + return new Rows(offset, count, null); + } + + static Rows of(int[] indices) { + return new Rows(0, indices.length, indices); + } + + int at(int i) { + if (indices == null) { + return offset + i; + } + return indices[i]; + } + } +} diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java index 73d70a295..2f6a2a54e 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java @@ -22,11 +22,11 @@ public record WriteSummary(long fileSize, long rowCount, Map /// One top-level column of the written file. /// - /// `stats` are the file statistics written into the footer: Rust's min, max, sum, null count - /// and NaN count, each folded over the `writeChunk` batches. A UTF-8 or binary bound is cut to - /// 64 bytes and then inexact (`minExact`/`maxExact` say so); a binary bound is a `byte[]`. A - /// statistic Rust does not keep for the column's type is `null`, as is every one of a column - /// without rows. + /// `stats` are the file statistics written into the footer: Rust's min, max, null count, NaN + /// count and uncompressed size, each accumulated over the `writeChunk` batches. A UTF-8 or binary + /// bound is cut to 64 bytes and always inexact (`minExact`/`maxExact` say so); a binary bound is + /// a `byte[]`. A statistic Rust does not keep for the column's type is `null`; a column without + /// rows has a zero null count and size. /// /// @param compressedSize the bytes of every segment of the column: data, zone map and /// dictionary, as Rust's `compressed_column_sizes` counts them diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java index 4471a5bf9..6052922a5 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java @@ -1,24 +1,20 @@ package io.github.dfa1.vortex.writer; -import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.DTypes; import io.github.dfa1.vortex.core.proto.ProtoScalarValue; import io.github.dfa1.vortex.writer.encode.NodeStats; +import io.github.dfa1.vortex.writer.encode.NullableData; import org.junit.jupiter.api.Test; -import java.math.BigDecimal; -import java.math.BigInteger; - import static org.assertj.core.api.Assertions.assertThat; -/// The fold rules of [FileStatsAccumulator] that `FileStatisticsInteropIntegrationTest` cannot reach -/// with a small file: sums that overflow only past millions of rows, and columns without any batch. +/// The accumulation rules of [FileStatsAccumulator], each on its own; `FileStatisticsInteropIntegrationTest` +/// checks them all together against vortex-jni. class FileStatsAccumulatorTest { @Test - void noBatch_carriesNoStat() { - // Given — Rust drops every per-batch stat of a column it never saw a batch of: even the null - // count and the sum, which a column with rows always has + void noBatch_zeroNullCountAndSize() { + // Given — Rust's accumulators start at zero and keep it without a batch FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.I64); sut.add(new long[0], 0); @@ -26,7 +22,7 @@ void noBatch_carriesNoStat() { NodeStats result = sut.stats(); // Then - assertThat(result).isEqualTo(NodeStats.NONE); + assertThat(result).isEqualTo(new NodeStats(null, null, null, null, null, 0L, 0L, null)); } @Test @@ -43,33 +39,71 @@ void variant_carriesNoStat() { } @Test - void unsignedSumOverflowAcrossBatches_dropsSum() { - // Given — each batch fits u64, their total does not: the fold is checked too - FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.U64); - sut.add(new long[]{-2L}, 1); - sut.add(new long[]{2L}, 1); + void utf8AllNullBatch_keepsTheOtherBatchesMax() { + // Given — Rust's BoundedMax skips a batch without a valid value + FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.UTF8.asNullable()); + sut.add(new NullableData(new String[]{"b", "a"}, new boolean[]{true, true}), 2); + sut.add(new NullableData(new String[]{null}, new boolean[]{false}), 1); // When NodeStats result = sut.stats(); - // Then — min and max are unaffected - assertThat(result.sum()).isNull(); - assertThat(result.max()).isEqualTo(ProtoScalarValue.ofUint64Value(-2L)); + // Then + assertThat(result.max()).isEqualTo(ProtoScalarValue.ofStringValue("b")); + assertThat(result.nullCount()).isEqualTo(1L); } @Test - void decimalSumPastMaxPrecision_dropsSum() { - // Given — at Rust's maximum precision a sum cannot widen: two values near 10^76 overflow it - BigDecimal big = new BigDecimal(BigInteger.TEN.pow(76).subtract(BigInteger.ONE)); - FileStatsAccumulator sut = new FileStatsAccumulator(DType.Decimal.of(76, 0, false)); - sut.add(new BigDecimal[]{big, big}, 2); + void utf8_boundsAreInexact() { + // Given — a short string is never cut, yet a bounded min/max only approximates Min/Max + FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.UTF8); + sut.add(new String[]{"a"}, 1); + + // When + boolean result = sut.boundsExact(); + + // Then + assertThat(result).isFalse(); + } + + @Test + void i64_boundsAreExact() { + // Given + FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.I64); + sut.add(new long[]{1, 2}, 2); + + // When + boolean result = sut.boundsExact(); + + // Then + assertThat(result).isTrue(); + } + + @Test + void integers_haveNoSum() { + // Given — Rust 0.91's default aggregates have no sum + FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.I64); + sut.add(new long[]{1, 2}, 2); // When NodeStats result = sut.stats(); // Then assertThat(result.sum()).isNull(); - assertThat(result.min()).isNotNull(); + } + + @Test + void uncompressedSize_addsUpTheBatches() { + // Given — 8 bytes a row; the first batch's one null adds a bitmap, the second is all valid + FileStatsAccumulator sut = new FileStatsAccumulator(DTypes.I64.asNullable()); + sut.add(new NullableData(new long[]{1, 0, 3}, new boolean[]{true, false, true}), 3); + sut.add(new NullableData(new long[]{4, 5}, new boolean[]{true, true}), 2); + + // When + NodeStats result = sut.stats(); + + // Then + assertThat(result.uncompressedSize()).isEqualTo(3 * 8 + 1 + 2 * 8L); } @Test @@ -82,6 +116,6 @@ void nullDtype_countsEveryRowNull() { NodeStats result = sut.stats(); // Then - assertThat(result).isEqualTo(new NodeStats(null, null, null, null, null, 4L, null, null, null)); + assertThat(result).isEqualTo(new NodeStats(null, null, null, null, null, 4L, 0L, null)); } } diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/UncompressedSizeTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/UncompressedSizeTest.java new file mode 100644 index 000000000..830da3e0b --- /dev/null +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/UncompressedSizeTest.java @@ -0,0 +1,284 @@ +package io.github.dfa1.vortex.writer; + +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.core.model.ExtensionId; +import io.github.dfa1.vortex.core.model.TimeUnit; +import io.github.dfa1.vortex.core.model.TimeUnitMetadata; +import io.github.dfa1.vortex.writer.encode.FixedSizeListData; +import io.github.dfa1.vortex.writer.encode.ListData; +import io.github.dfa1.vortex.writer.encode.ListViewData; +import io.github.dfa1.vortex.writer.encode.NullableData; +import io.github.dfa1.vortex.writer.encode.StructData; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; + +import java.math.BigDecimal; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/// Rust's `uncompressed_size_in_bytes` of one batch, rule by rule; `FileStatisticsInteropIntegrationTest` +/// checks the same sizes against vortex-jni. +class UncompressedSizeTest { + + @ParameterizedTest + @CsvSource({ + "4, 4, 0", // every row valid: no validity + "0, 4, 2", // no row valid: a ConstantArray(false)'s 2-byte scalar + "3, 4, 1", // some valid: a packed bitmap + "8, 9, 2", + "0, 0, 0"}) + void validityBytes(long valid, long rows, long expected) { + // Given / When + long result = UncompressedSize.validityBytes(valid, rows); + + // Then + assertThat(result).isEqualTo(expected); + } + + @Nested + class Flat { + + @Test + void primitive_widthPerRow() { + // Given / When + long result = UncompressedSize.of(DTypes.I32, new int[5], 5); + + // Then + assertThat(result).isEqualTo(20); + } + + @Test + void nullablePrimitive_addsItsBitmap() { + // Given + NullableData data = new NullableData(new long[3], new boolean[]{true, false, true}); + + // When + long result = UncompressedSize.of(DTypes.I64.asNullable(), data, 3); + + // Then + assertThat(result).isEqualTo(3 * 8 + 1); + } + + @Test + void bool_packedBits() { + // Given / When + long result = UncompressedSize.of(DTypes.BOOL, new boolean[9], 9); + + // Then + assertThat(result).isEqualTo(2); + } + + @Test + void decimal_widthOfItsPrecision() { + // Given — precision 10 takes 8 bytes, whatever the values + DType.Decimal dtype = DType.Decimal.of(10, 2, false); + + // When + long result = UncompressedSize.of(dtype, new BigDecimal[]{BigDecimal.ONE, BigDecimal.TEN}, 2); + + // Then + assertThat(result).isEqualTo(16); + } + + @Test + void utf8_viewsPlusValuesTooLongToInline() { + // Given — 12 bytes inline, 13 do not; the null row's value is not counted + String[] values = {"a".repeat(12), "b".repeat(13), "c".repeat(20)}; + NullableData data = new NullableData(values, new boolean[]{true, true, false}); + + // When + long result = UncompressedSize.of(DTypes.UTF8.asNullable(), data, 3); + + // Then + assertThat(result).isEqualTo(3 * 16 + 13 + 1); + } + + @Test + void utf8_countsUtf8Bytes() { + // Given — 7 two-byte characters: 14 bytes, too long to inline + String[] values = {"é".repeat(7)}; + + // When + long result = UncompressedSize.of(DTypes.UTF8, values, 1); + + // Then + assertThat(result).isEqualTo(16 + 14); + } + + @Test + void binary_viewsPlusValuesTooLongToInline() { + // Given + byte[][] values = {new byte[2], new byte[40]}; + + // When + long result = UncompressedSize.of(DTypes.BINARY, values, 2); + + // Then + assertThat(result).isEqualTo(2 * 16 + 40); + } + + @Test + void null_isZero() { + // Given / When + long result = UncompressedSize.of(DTypes.NULL, new long[7], 7); + + // Then + assertThat(result).isZero(); + } + + @Test + void extension_isItsStorage() { + // Given + DType.Extension date = new DType.Extension(ExtensionId.VORTEX_DATE, DTypes.I32, + new TimeUnitMetadata(TimeUnit.Days), false); + + // When + long result = UncompressedSize.of(date, new int[3], 3); + + // Then + assertThat(result).isEqualTo(12); + } + } + + @Nested + class Nesting { + + @Test + void struct_sumsItsFieldsAndAddsItsValidity() { + // Given + DType.Struct dtype = new DType.Struct(List.of(ColumnName.of("a"), ColumnName.of("b")), + List.of(DTypes.I32, DTypes.I8), true); + NullableData data = new NullableData(new StructData(List.of(new int[4], new byte[4])), + new boolean[]{true, false, true, true}); + + // When + long result = UncompressedSize.of(dtype, data, 4); + + // Then + assertThat(result).isEqualTo(4 * 4 + 4 + 1); + } + + @Test + void list_elementsOfValidListsPlusViews() { + // Given — a null list holding no element: the valid lists tile [0, 3) + DType.List dtype = new DType.List(DTypes.I32, true); + ListData lists = new ListData(new int[3], new long[]{0, 2, 2, 3}, 3); + NullableData data = new NullableData(lists, new boolean[]{true, false, true}); + + // When + long result = UncompressedSize.of(dtype, data, 3); + + // Then — 3 elements, 16 bytes of offset and size per row, a bitmap + assertThat(result).isEqualTo(3 * 4 + 3 * 16 + 1); + } + + @Test + void list_nullListsElementsAreNotCounted() { + // Given — the null list spans elements [2, 5) + DType.List dtype = new DType.List(DTypes.I32, true); + ListData lists = new ListData(new int[6], new long[]{0, 2, 5, 6}, 3); + NullableData data = new NullableData(lists, new boolean[]{true, false, true}); + + // When + long result = UncompressedSize.of(dtype, data, 3); + + // Then + assertThat(result).isEqualTo(3 * 4 + 3 * 16 + 1); + } + + @Test + void list_ofStrings_sizesTheUsedElements() { + // Given + DType.List dtype = new DType.List(DTypes.UTF8, false); + ListData lists = new ListData(new String[]{"short", "x".repeat(20), "y"}, new long[]{0, 2, 3}, 2); + + // When + long result = UncompressedSize.of(dtype, lists, 2); + + // Then + assertThat(result).isEqualTo(3 * 16 + 20 + 2 * 16); + } + + @Test + void list_noValidList_onlyItsViews() { + // Given + DType.List dtype = new DType.List(DTypes.I32, true); + NullableData data = new NullableData(new ListData(new int[2], new long[]{0, 2}, 1), new boolean[]{false}); + + // When + long result = UncompressedSize.of(dtype, data, 1); + + // Then + assertThat(result).isEqualTo(16 + 2); + } + + @Test + void listView_overlappingLists_countEachUse() { + // Given — both lists use element 1: three uses of a two-element run + DType.List dtype = new DType.List(DTypes.I64, false); + ListViewData lists = new ListViewData(new long[2], new int[]{0, 1}, new int[]{2, 1}, 2); + + // When + long result = UncompressedSize.of(dtype, lists, 2); + + // Then + assertThat(result).isEqualTo(3 * 8 + 2 * 16); + } + + @Test + void listView_gapsAndOverlapsThatTile_sizeTheRunAsASlice() { + // Given — [0, 2), [1, 2) and [3, 4) use 4 elements and span [0, 4): Rust slices the run + DType.List dtype = new DType.List(DTypes.UTF8, false); + String[] elements = {"a", "b", "c".repeat(30), "d"}; + ListViewData lists = new ListViewData(elements, new int[]{0, 1, 3}, new int[]{2, 1, 1}, 3); + + // When + long result = UncompressedSize.of(dtype, lists, 3); + + // Then — the slice holds "ccc…", which no list uses + assertThat(result).isEqualTo(4 * 16 + 30 + 3 * 16); + } + + @Test + void fixedSizeList_everyElementNullRowsToo() { + // Given + DType.FixedSizeList dtype = new DType.FixedSizeList(DTypes.I16, 3, true); + NullableData data = new NullableData(new FixedSizeListData(new short[6], 2), new boolean[]{false, true}); + + // When + long result = UncompressedSize.of(dtype, data, 2); + + // Then + assertThat(result).isEqualTo(6 * 2 + 1); + } + + @Test + void map_keysAndValuesOfValidEntries() { + // Given + DType.Map dtype = new DType.Map(DTypes.I32, DTypes.I64, false, false); + ListViewData entries = new ListViewData(new StructData(List.of(new int[3], new long[3])), + new int[]{0, 2}, new int[]{2, 1}, 2); + + // When + long result = UncompressedSize.of(dtype, entries, 2); + + // Then + assertThat(result).isEqualTo(3 * (4 + 8) + 2 * 16); + } + } + + @Test + void supports_notVariant() { + // Given + DType.Struct dtype = new DType.Struct(List.of(ColumnName.of("v")), List.of(DTypes.VARIANT), false); + + // When / Then + assertThat(UncompressedSize.supports(dtype)).isFalse(); + assertThat(UncompressedSize.supports(DTypes.I64)).isTrue(); + } +} diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java index feebc8f93..a20f67413 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java @@ -808,7 +808,7 @@ void summarizesTheFile() throws IOException { assertThat(result.rowCount()).isEqualTo(4); assertThat(result.columns()).containsOnlyKeys(ColumnName.of("id"), ColumnName.of("value")); ArrayStats id = result.columns().get(ColumnName.of("id")).stats(); - assertThat(id).isEqualTo(new ArrayStats(1L, true, 9L, true, 15L, null, null, null, 0L, null, null)); + assertThat(id).isEqualTo(new ArrayStats(1L, true, 9L, true, null, null, null, null, 0L, 4 * 8L, null)); assertThat(result.columns().get(ColumnName.of("value")).compressedSize()).isPositive(); } @@ -843,8 +843,8 @@ void writeChunkAfterFinish_throws() throws IOException { } @Test - void noRows_hasEmptyStats() throws IOException { - // Given — Rust keeps no statistic of a column it saw no batch of + void noRows_zeroNullCountAndSize() throws IOException { + // Given — Rust's null count and size start at zero; a float column also counts NaNs VortexWriter sut = VortexWriter.create(java.nio.channels.Channels.newChannel(out), SCHEMA, WriteOptions.defaults()); // When @@ -852,8 +852,9 @@ void noRows_hasEmptyStats() throws IOException { // Then assertThat(result.rowCount()).isZero(); - assertThat(result.columns().values()).extracting(WriteSummary.Column::stats) - .containsOnly(ArrayStats.empty()); + assertThat(result.columns().values()).extracting(WriteSummary.Column::stats).containsExactly( + new ArrayStats(null, false, null, false, null, null, null, null, 0L, 0L, null), + new ArrayStats(null, false, null, false, null, null, null, null, 0L, 0L, 0L)); } } }