diff --git a/CHANGELOG.md b/CHANGELOG.md index 20bf9f096..6c86c0bdf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - `EncodeContext` has a new last component, `trace`, so code calling its canonical constructor passes `EncodeTrace.NONE`. ([#601](https://github.com/dfa1/vortex-java/issues/601)) +### Removed +- `CompressorBuilder#seed(long)` of the `vortex-fsst` module: training takes Rust's fixed sample, not a seeded one. ([#583](https://github.com/dfa1/vortex-java/issues/583)) + ### Fixed +- FSST symbol tables are trained as Rust's `fsst-rs` does (its sample, generations, candidate gains and cut), so a text column without a null gets Rust's FSST encoding up to ties of equal gain, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650)) - A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696)) diff --git a/docs/compatibility.md b/docs/compatibility.md index 12fb0ddde..839b8ff3a 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -130,7 +130,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul | `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) | | `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct | | `vortex.chunked` | `ChunkedEncodingDecoder` | `ChunkedEncodingEncoder` | ✅ | ✅ | Primitive + Struct concat | -| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary | +| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary. The symbol table is trained as Rust's `fsst-rs` v0.6.0 does ([#583](https://github.com/dfa1/vortex-java/issues/583)): its sample, five generations, candidate gains, cut and final code order. Not byte for byte Rust's, by decision: candidates of equal gain and length pop in the order they were counted, where Rust pops them in its hash map's iteration order, which depends on the CPU's SIMD width (Rust's own table differs between x86 and arm); about 90% of the symbols and a code stream within 1% are Rust's. A column with nulls trains on its empty placeholder rows, where Rust trains on the valid rows only, until its validity lives in the encoding (#650) | | `vortex.list` | `ListEncodingDecoder` | `ListEncodingEncoder` | ✅ | ✅ | A nullable list's validity is its optional third child, as in Rust | | `vortex.listview` | `ListViewEncodingDecoder` | `ListViewEncodingEncoder` | ✅ | ✅ | Validity lives in the encoding's own fourth child slot, not under a `vortex.masked` wrapper. Read: both shapes decode. Write: only `vortex.map`'s entries child takes this path — a plain nullable `DType.List` column still routes through `vortex.masked` + `vortex.list` | | `vortex.map` | `MapEncodingDecoder` | `MapEncodingEncoder` | ✅ | ✅ | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately | diff --git a/docs/reference.md b/docs/reference.md index 53be0ba2f..5935f3ae4 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -423,9 +423,9 @@ zero-ceremony standalone use, and both paths produce identical output for the sa | Method | Notes | |-------------------------|-----------------------------------------------------------------------------| -| `new CompressorBuilder()` | Fresh builder using a fixed reproducible default seed | -| `seed(long)` | Override the training-sample seed; returns `this` for chaining | -| `train(byte[][] rows)` | Runs the bottom-up generation loop, returns a trained `Compressor` (deterministic in rows + seed) | +| `new CompressorBuilder()` | A builder; training takes no options: it follows Rust's `fsst-rs`, its sample included | +| `train(byte[][] rows)` | Runs the bottom-up generation loop, returns a trained `Compressor` (deterministic in the rows) | +| `train(byte[] rowBytes, int[] rowOffsets, int rowCount)` | The same over rows held contiguously, as a varbin column has them | ### `Compressor` @@ -437,7 +437,7 @@ Immutable trained symbol table (up to 255 codes; `0xFF` is the escape). Produced | `symbolCount()` | Number of symbols in the table (0–255) | | `packedSymbol(int code)` | The symbol's bytes packed LSB-first into a `long` | | `symbolLength(int code)` | The symbol's length in bytes (1–8) | -| `codesSortedByLength()` | Code permutation, multi-byte length-ascending then all length-1 last | +| `codesSortedByLength()` | Code permutation in Rust's final order: 2-byte symbols no longer symbol starts with, those one does (last taken first), 3–8 bytes by length, then the 1-byte ones | | `toDecompressor()` | A `Decompressor` bound to this table | | `compress(byte[] in, int start, int end, byte[] out, long outPos)` | Greedy longest-match compress; size `out` ≥ `2 * (end - start)` | | `compress(MemorySegment in, long start, long end, MemorySegment out, long outPos)` | FFM-native hot path; same sizing rule | diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java index f9a9a9249..5ef1c89c7 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java @@ -36,12 +36,27 @@ public final class Compressor { static final VarHandle LONG_LE_BYTES = MethodHandles.byteArrayViewVarHandle(long[].class, ByteOrder.LITTLE_ENDIAN); + /// Slots of the symbol table Rust's builder keeps, whose real symbols are the first `symbolCount()`. + private static final int SYMBOL_SLOTS = 255; + private final List symbolsByGainDescending; private final Matcher matcher; + /// The bytes of each of the table's [#SYMBOL_SLOTS] slots as Rust's builder holds them: the real + /// symbols, then, past them, whatever an earlier generation left there, which Rust never clears. + private final long[] slots; - private Compressor(List symbolsByGainDescending, Matcher matcher) { + private Compressor(List symbolsByGainDescending, Matcher matcher, long[] slots) { this.symbolsByGainDescending = symbolsByGainDescending; this.matcher = matcher; + this.slots = slots; + } + + private static long[] slotsOf(List symbols, long[] previous) { + long[] slots = previous.clone(); + for (int code = 0; code < symbols.size(); code++) { + slots[code] = symbols.get(code).packedBytes(); + } + return slots; } /// Creates a compressor over the given symbols, which must be in gain-descending order (code = @@ -51,7 +66,7 @@ private Compressor(List symbolsByGainDescending, Matcher matcher) { /// @return a compressor bound to that table static Compressor of(List symbolsByGainDescending) { List copy = List.copyOf(symbolsByGainDescending); - return new Compressor(copy, Matcher.of(copy)); + return new Compressor(copy, Matcher.of(copy), slotsOf(copy, new long[SYMBOL_SLOTS])); } /// Rebuilds a compressor from new symbols, reusing `previous`'s [Matcher] backing arrays instead @@ -64,7 +79,7 @@ static Compressor of(List symbolsByGainDescending) { /// matcher arrays static Compressor rebuild(List symbolsByGainDescending, Compressor previous) { List copy = List.copyOf(symbolsByGainDescending); - return new Compressor(copy, Matcher.rebuild(copy, previous.matcher)); + return new Compressor(copy, Matcher.rebuild(copy, previous.matcher), slotsOf(copy, previous.slots)); } /// Returns an equivalent compressor whose codes are permuted: the symbol currently at code @@ -119,49 +134,70 @@ public int symbolLength(int code) { return symbolsByGainDescending.get(code).length(); } - /// Returns a permutation of the codes ordered by symbol length: all multi-byte symbols (length - /// 2-8) first in non-decreasing length order, then all length-1 symbols last. + /// Returns a permutation of the codes in the order Rust's `fsst-rs` numbers them once its table is + /// final (`finalize`): the 2-byte symbols that no other symbol shares its first two bytes with, in + /// the order they were taken; then the 2-byte symbols some longer symbol starts with, last taken + /// first; then the 3, 4, ..., 8-byte symbols, each length in the order they were taken; then the + /// 1-byte symbols. /// - /// This is pure, wire-format-agnostic reordering logic — "sort by length, single-byte-last" — - /// with no `vortex.fsst`-specific naming or semantics baked in. The `vortex.fsst` wire adapter - /// (a separate module) happens to interpret this exact permutation as its on-wire symbol order, - /// but that interpretation lives entirely in the adapter; the algorithm module only knows how to - /// sort. Ties within a length bucket keep their gain-descending relative order (a stable sort), - /// so the permutation is deterministic. + /// The split of the 2-byte symbols lets Rust's compressor skip its hash table for a 2-byte code + /// below the split, which no longer symbol can extend. Rust asks "does another symbol start with + /// these two bytes" of every slot of its table, including the ones an earlier training generation + /// left behind past the final symbols, so a table smaller than an earlier one splits differently + /// than its own symbols alone would. /// - /// @return an `int[]` permutation of `0 .. symbolCount() - 1` in length-sorted, single-byte-last - /// order + /// @return an `int[]` permutation of `0 .. symbolCount() - 1`: the code at each position public int[] codesSortedByLength() { - // Counting sort over symbol length: lengths are bounded to 1-8 (Matcher.maxSymbolLength()), - // so a comparator-based sort is unnecessary work — this is O(symbolCount) with no boxing, - // and a single forward pass over codes 0..symbolCount()-1 (their gain-descending order) into - // each length's bucket is stable by construction, with no comparator needed to break ties. int maxSymbolLength = Matcher.maxSymbolLength(); - int[] countByLength = new int[maxSymbolLength + 1]; // index 1..maxSymbolLength; 0 unused int n = symbolCount(); + int[] countByLength = new int[maxSymbolLength + 1]; // index 1..maxSymbolLength; 0 unused for (int code = 0; code < n; code++) { countByLength[symbolLength(code)]++; } - // Bucket start offsets in wire order: multi-byte lengths 2..maxSymbolLength ascending, then - // length-1 last. - int[] bucketStart = new int[maxSymbolLength + 1]; - int offset = 0; - for (int length = 2; length <= maxSymbolLength; length++) { - bucketStart[length] = offset; - offset += countByLength[length]; + // First position of each length: 1-byte symbols after all the others; the 2-byte ones from 0, + // split at the suffix limit; then each longer length after the one before. + int[] nextPosition = new int[maxSymbolLength + 1]; + nextPosition[1] = n - countByLength[1]; + nextPosition[2] = 0; + for (int length = 2; length < maxSymbolLength; length++) { + nextPosition[length + 1] = nextPosition[length] + countByLength[length]; } - bucketStart[1] = offset; + int noSuffix = 0; + int hasSuffix = nextPosition[3]; int[] result = new int[n]; - int[] nextSlot = bucketStart.clone(); for (int code = 0; code < n; code++) { int length = symbolLength(code); - result[nextSlot[length]++] = code; + int position; + if (length == 2) { + if (sharesFirstTwoBytes(code)) { + hasSuffix--; + position = hasSuffix; + } else { + position = noSuffix++; + } + } else { + position = nextPosition[length]++; + } + result[position] = code; } return result; } + /// Whether another slot starts with the same two bytes as the 2-byte symbol `code`: as Rust's + /// `finalize` asks it, of every slot of its table, the ones past the real symbols too, with what + /// an earlier generation left in them. + private boolean sharesFirstTwoBytes(int code) { + long first2 = slots[code] & 0xFFFF; + for (int other = 0; other < SYMBOL_SLOTS; other++) { + if (other != code && (slots[other] & 0xFFFF) == first2) { + return true; + } + } + return false; + } + /// Builds a [Decompressor] over this compressor's table so the same trained symbols round-trip /// without re-deriving the parallel arrays at the call site. /// diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java index 8b3acee6f..5972ca648 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/CompressorBuilder.java @@ -2,49 +2,22 @@ import java.util.List; -/// Trains an FSST [Compressor] from a corpus of rows using the FSST paper's bottom-up algorithm -/// (Algorithm 3) plus the Rust reference's four engineering refinements (issue #287). +/// Trains an FSST [Compressor] from a corpus of rows as Rust's `fsst-rs` does (`Compressor::train`, +/// the FSST paper's Algorithm 3 with the C++ reference's refinements): the same sample, the same five +/// generations over a growing share of its lines, and the same candidates, gains and cut. /// -/// Training runs [#GENERATIONS] generations starting from an empty symbol table. Each generation -/// compress-counts a growing fraction of one fixed [Sample] with the current table, then rebuilds -/// the table from the top-gain candidates (see [TrainingGeneration]). The refinements — adaptive -/// per-generation sample fraction, per-generation min-count pruning, the 8x single-byte gain boost, -/// and a final cost-based prune — are all applied inside that per-generation step; this class only -/// drives the loop and threads the seed through. -/// -/// Training is deterministic: the same rows and seed always produce a byte-identical [Compressor], -/// so an encoder built on top of it produces byte-identical output for a given input. Use the -/// builder to override the seed: +/// Training is deterministic: the same rows always produce a byte-identical [Compressor]. Rust's own +/// tables differ from this one only in where candidates of exactly equal gain and length are +/// ordered, which depends on its hash map ([TrainingGeneration]). /// /// ```java -/// Compressor compressor = new CompressorBuilder().seed(42L).train(rows); +/// Compressor compressor = new CompressorBuilder().train(rows); /// ``` public final class CompressorBuilder { - /// Number of training generations. The FSST paper reports the table stabilizing after roughly - /// five passes; each pass can pair adjacent symbols to roughly double their length, so five - /// passes grow symbols to the full 8-byte length from an empty start. - private static final int GENERATIONS = Sample.SAMPLE_FRACTION_NUMERATORS.length; - - /// Default PRNG seed when [#seed(long)] is not called. A fixed constant (not wall-clock derived) - /// keeps training reproducible out of the box. - private static final long DEFAULT_SEED = 0x5EEDL; - - private long seed = DEFAULT_SEED; - - /// Creates a builder using the default seed until [#seed(long)] is called. + /// Creates a builder. public CompressorBuilder() { - // Explicit no-arg constructor: fields default to the reproducible DEFAULT_SEED. - } - - /// Sets the PRNG seed used to draw the training sample. The same rows and seed always train a - /// byte-identical compressor. - /// - /// @param seed the sample-drawing seed - /// @return this builder, for chaining - public CompressorBuilder seed(long seed) { - this.seed = seed; - return this; + // Explicit no-arg constructor: nothing to configure, Rust's training takes no options. } /// Trains a compressor over `rows`, running the full bottom-up generation loop. @@ -56,38 +29,37 @@ public CompressorBuilder seed(long seed) { /// empty) /// @return the trained compressor public Compressor train(byte[][] rows) { - return trainOn(Sample.draw(rows, seed)); + if (rows.length == 0) { + return Compressor.of(List.of()); + } + return trainOn(Sample.draw(rows)); } /// Trains a compressor over rows held contiguously, where row `i` spans /// `rowBytes[rowOffsets[i], rowOffsets[i + 1])`. /// - /// Equivalent to [#train(byte[][])] — same sample, same resulting table for the same seed — - /// but takes the corpus in the contiguous shape a varbin column already has, so the caller - /// need not materialize one `byte[]` per row just to train. + /// Equivalent to [#train(byte[][])] (same sample, same table) but takes the corpus in the + /// contiguous shape a varbin column already has, so the caller need not materialize one `byte[]` + /// per row just to train. /// /// @param rowBytes all rows' bytes concatenated /// @param rowOffsets `rowCount + 1` cumulative offsets into `rowBytes` /// @param rowCount the number of rows /// @return the trained compressor public Compressor train(byte[] rowBytes, int[] rowOffsets, int rowCount) { - return trainOn(Sample.draw(rowBytes, rowOffsets, rowCount, seed)); + if (rowCount == 0) { + return Compressor.of(List.of()); + } + return trainOn(Sample.draw(rowBytes, rowOffsets, rowCount)); } private Compressor trainOn(Sample sample) { Compressor compressor = Compressor.of(List.of()); - if (sample.chunkCount() == 0) { - return compressor; - } // Reused across every generation instead of allocating fresh count arrays five times over // (issue #395). TrainingGeneration.Counts counts = new TrainingGeneration.Counts(); - for (int gen = 0; gen < GENERATIONS; gen++) { - boolean finalGeneration = gen == GENERATIONS - 1; - int chunkLimit = sample.chunkCountForGeneration(gen); - int fractionNumerator = Sample.SAMPLE_FRACTION_NUMERATORS[gen]; - List symbols = TrainingGeneration.run( - compressor, sample, chunkLimit, fractionNumerator, finalGeneration, counts); + for (int fraction : Sample.FRACTION_NUMERATORS) { + List symbols = TrainingGeneration.run(compressor, sample, fraction, counts); // Reuse compressor's own (now fully consumed) matcher arrays instead of allocating a // fresh ~288 KB pair of tables on every one of the five generations (issue #393 #7). compressor = Compressor.rebuild(symbols, compressor); diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/Matcher.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/Matcher.java index 608b4f11a..25a671521 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/Matcher.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/Matcher.java @@ -49,7 +49,7 @@ public static Matcher of(List symbolsByGainDescending) { } /// Rebuilds a matcher from new symbols, re-seeding `previous`'s backing arrays in place instead - /// of allocating fresh ~288 KB tables. Training's per-generation loop calls this six times per + /// of allocating fresh ~288 KB tables. Training's per-generation loop calls this five times per /// [CompressorBuilder#train(byte[][])]; without reuse each call allocates and zero-fills a fresh /// [LossyPerfectHashTable] and [ShortCodeTable] that the very next call immediately discards /// (issue #393 #7). diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/Sample.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/Sample.java index aa4d0c366..84d04d372 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/Sample.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/Sample.java @@ -1,78 +1,48 @@ package io.github.dfa1.vortex.fsst; -import java.util.Random; - -/// A fixed, byte-size-bounded training sample drawn deterministically from a corpus of rows. -/// -/// FSST training does not scan the whole input on every generation. Instead it draws one bounded -/// sample up front — targeting [#TARGET_SAMPLE_BYTES] bytes assembled from contiguous chunks of at -/// most [#MAX_CHUNK_BYTES] bytes, each taken from a randomly chosen row — and then each of the five -/// training generations replays only a growing fraction of that same sample. This replaces the old -/// `FsstEncodingEncoder`'s string-count-bounded, always-full-sample scheme, which on long-string -/// columns pulled in far more training bytes than the reference intends for no compression benefit -/// (issue #287). Sampling by bytes keeps training cost proportional to the sample size regardless of -/// how long individual rows are. +/// The training sample, drawn as Rust's `fsst-rs` does (`builder.rs`'s `make_sample`, itself the C++ +/// reference's): the rows as they are when their bytes total under [#TARGET_SAMPLE_BYTES], else +/// [#LINE_BYTES]-byte lines of randomly picked non-empty rows until the target is reached. The +/// pick is a fixed hash sequence ([#hash(long)]), not a seeded generator, so a given corpus always +/// draws the same sample, and the same one Rust draws. /// -/// The sample is held as a single concatenated byte array with explicit chunk boundaries, because a -/// symbol must never be counted across a chunk boundary: chunks come from unrelated rows, so a -/// "symbol" spanning two chunks would be an artifact of sampling, not a real substring of any row. -/// -/// Drawing is deterministic given a seed: the same rows and seed always produce a byte-identical -/// sample, which is what makes the whole trained table reproducible across runs (the encoder's -/// output must be byte-identical for a given input). +/// A line is a slice of one row, and a short corpus keeps its empty rows as empty lines: training +/// skips lines by their index, so the indices have to be Rust's too. final class Sample { - /// Target total sample size in bytes. The Rust reference `spiraldb/fsst` targets ~16KB; a - /// sample this size is large enough to learn a stable table while keeping every generation's - /// compress-count pass cheap. - static final int TARGET_SAMPLE_BYTES = 16 * 1024; + /// A corpus of at least this many bytes is sampled down to about this many: Rust's + /// `FSST_SAMPLETARGET`. + static final int TARGET_SAMPLE_BYTES = 1 << 14; - /// Maximum length of a single contiguous chunk drawn from one row. Long rows contribute several - /// chunks; short rows contribute one (truncated to the row length). Bounding chunk length keeps - /// the sample spread across many rows rather than dominated by a single long one. - static final int MAX_CHUNK_BYTES = 512; + /// Longest line a sample takes from one row: Rust's `FSST_SAMPLELINE`. + static final int LINE_BYTES = 512; - /// Denominator for the per-generation growing sample fraction. Each generation replays - /// `SAMPLE_FRACTION_NUMERATORS[gen] / SAMPLE_FRACTION_DENOMINATOR` of the sample's chunks. - static final int SAMPLE_FRACTION_DENOMINATOR = 128; + /// Rust's seed of the sample's hash sequence. + private static final long SAMPLE_SEED = 4637947L; - /// Per-generation numerators over [#SAMPLE_FRACTION_DENOMINATOR]. Early generations see only a - /// small fraction of the sample (the table is still small and low-quality, so compressing more - /// bytes buys nothing), ramping to the full sample on the last generation once the table has - /// mostly converged. These are the Rust reference's fractions: ~6%, ~30%, ~53%, ~77%, 100%. - static final int[] SAMPLE_FRACTION_NUMERATORS = {8, 38, 68, 98, 128}; + /// The generations' sample fractions, out of [#FRACTION_DENOMINATOR]: Rust's `GENERATIONS`. + static final int[] FRACTION_NUMERATORS = {8, 38, 68, 98, 128}; - private final byte[] bytes; + /// The denominator of the sample fractions. + static final int FRACTION_DENOMINATOR = 128; - /// Chunk boundaries as one array of `chunkCount + 1` cumulative offsets: chunk `i` spans - /// `[chunkBoundaries[i], chunkBoundaries[i + 1])`. Chunks are written back-to-back in - /// [#draw(byte[][], long)], so a chunk's end always equals the next chunk's start — a separate - /// ends array would just duplicate `chunkBoundaries[1..]`. The array may be over-allocated - /// (sized for the worst case of every chunk being 1 byte); only indices `0 .. chunkCount` are - /// valid, so `chunkCount` is tracked separately rather than trimming the array to size. - private final int[] chunkBoundaries; - private final int chunkCount; + private final byte[] bytes; + private final int[] lineBoundaries; + private final int lineCount; + private final boolean sampled; - private Sample(byte[] bytes, int[] chunkBoundaries, int chunkCount) { + private Sample(byte[] bytes, int[] lineBoundaries, int lineCount, boolean sampled) { this.bytes = bytes; - this.chunkBoundaries = chunkBoundaries; - this.chunkCount = chunkCount; + this.lineBoundaries = lineBoundaries; + this.lineCount = lineCount; + this.sampled = sampled; } - /// Draws a deterministic byte-size-bounded sample from `rows`. - /// - /// Rows are selected at random; each selection contributes one contiguous chunk of at most - /// [#MAX_CHUNK_BYTES] bytes, starting at a random offset within the chosen row. Empty rows are - /// skipped. Drawing stops once [#TARGET_SAMPLE_BYTES] bytes have been collected, so a corpus - /// smaller than the target is used in full. When every row is empty the sample is empty. + /// Draws the sample of rows held one array each. /// - /// @param rows the corpus rows, each a raw byte array (never `null`, but individually may be - /// empty) - /// @param seed the PRNG seed; the same rows and seed yield a byte-identical sample - /// @return a sample of up to [#TARGET_SAMPLE_BYTES] bytes split into per-row chunks - @SuppressWarnings("java:S2245") // Deterministic PRNG is the contract: training samples must be - // reproducible across builds. No security boundary here. - static Sample draw(byte[][] rows, long seed) { + /// @param rows the corpus rows, never `null`; a row may be empty + /// @return the sample + static Sample draw(byte[][] rows) { return draw(rows.length, new RowAccess() { @Override public int rowLength(int row) { @@ -80,27 +50,20 @@ public int rowLength(int row) { } @Override - public void copyChunk(int row, int start, int len, byte[] out, int outPos) { + public void copy(int row, int start, int len, byte[] out, int outPos) { System.arraycopy(rows[row], start, out, outPos, len); } - }, seed); + }); } - /// Draws a deterministic byte-size-bounded sample from rows held contiguously, where row `i` - /// spans `rowBytes[rowOffsets[i], rowOffsets[i + 1])`. - /// - /// Behaves exactly like [#draw(byte[][], long)] — same selection order, same chunk lengths, - /// same bytes for the same seed — but reads its rows out of one shared array instead of one - /// `byte[]` object per row, so a caller holding its corpus contiguously never has to explode it - /// into per-row arrays just to train. + /// Draws the sample of rows held contiguously, row `i` spanning `rowBytes[rowOffsets[i], + /// rowOffsets[i + 1])`. /// /// @param rowBytes all rows' bytes concatenated /// @param rowOffsets `rowCount + 1` cumulative offsets into `rowBytes` /// @param rowCount the number of rows - /// @param seed the PRNG seed; the same rows and seed yield a byte-identical sample - /// @return a sample of up to [#TARGET_SAMPLE_BYTES] bytes split into per-row chunks - @SuppressWarnings("java:S2245") // Deterministic PRNG is the contract, as in the byte[][] overload. - static Sample draw(byte[] rowBytes, int[] rowOffsets, int rowCount, long seed) { + /// @return the sample + static Sample draw(byte[] rowBytes, int[] rowOffsets, int rowCount) { return draw(rowCount, new RowAccess() { @Override public int rowLength(int row) { @@ -108,119 +71,111 @@ public int rowLength(int row) { } @Override - public void copyChunk(int row, int start, int len, byte[] out, int outPos) { + public void copy(int row, int start, int len, byte[] out, int outPos) { System.arraycopy(rowBytes, rowOffsets[row] + start, out, outPos, len); } - }, seed); + }); } - /// Shared sampling core behind both `draw` overloads: identical row-selection and - /// chunk-copying logic, addressed through `access` so it works whether rows live as one - /// `byte[]` per row or concatenated in a shared buffer. - /// - /// @param rowCount the number of rows - /// @param access row-length and chunk-copy operations for the caller's row storage - /// @param seed the PRNG seed; the same rows and seed yield a byte-identical sample - /// @return a sample of up to [#TARGET_SAMPLE_BYTES] bytes split into per-row chunks - private static Sample draw(int rowCount, RowAccess access, long seed) { + private static Sample draw(int rowCount, RowAccess access) { long totalBytes = 0; - // Pre-filter non-empty row indices once, up front: drawing then picks uniformly among only - // these `nonEmptyRows` candidates, an O(1) selection per chunk. The alternative — drawing a - // row index from the full row-count range and re-drawing on an empty hit — is an unbounded - // rejection sampler whose expected draw count is quadratic in the corpus's sparsity (e.g. a - // 1M-row, 0.01%-non-empty corpus), not the bounded skip a prior version of this comment - // claimed. - int[] nonEmptyRowIndices = new int[rowCount]; - int nonEmptyRows = 0; + for (int i = 0; i < rowCount; i++) { + totalBytes += access.rowLength(i); + } + if (totalBytes < TARGET_SAMPLE_BYTES) { + return whole(rowCount, access, (int) totalBytes); + } + return sampled(rowCount, access); + } + + private static Sample whole(int rowCount, RowAccess access, int totalBytes) { + byte[] out = new byte[totalBytes]; + int[] boundaries = new int[rowCount + 1]; + int pos = 0; for (int i = 0; i < rowCount; i++) { int len = access.rowLength(i); - totalBytes += len; - if (len > 0) { - nonEmptyRowIndices[nonEmptyRows++] = i; - } + access.copy(i, 0, len, out, pos); + pos += len; + boundaries[i + 1] = pos; } - if (nonEmptyRows == 0) { - return new Sample(new byte[0], new int[]{0}, 0); + return new Sample(out, boundaries, rowCount, false); + } + + private static Sample sampled(int rowCount, RowAccess access) { + // a line is at most LINE_BYTES, and the draw stops at the first line past the target + byte[] out = new byte[TARGET_SAMPLE_BYTES + LINE_BYTES]; + int[] boundaries = new int[TARGET_SAMPLE_BYTES + 2]; + int lines = 0; + int size = 0; + long rnd = hash(SAMPLE_SEED); + while (size < TARGET_SAMPLE_BYTES) { + rnd = hash(rnd); + int start = (int) Long.remainderUnsigned(rnd, rowCount); + int row = firstNonEmptyFrom(start, rowCount, access); + if (row < 0) { + break; + } + int rowLength = access.rowLength(row); + int chunks = 1 + (rowLength - 1) / LINE_BYTES; + rnd = hash(rnd); + int chunk = LINE_BYTES * (int) Long.remainderUnsigned(rnd, chunks); + int len = Math.min(LINE_BYTES, rowLength - chunk); + access.copy(row, chunk, len, out, size); + size += len; + lines++; + boundaries[lines] = size; } + return new Sample(out, boundaries, lines, true); + } - int target = (int) Math.min(TARGET_SAMPLE_BYTES, totalBytes); - byte[] out = new byte[target]; - // Every chunk is at least 1 byte, so `target` chunks is the worst-case count (plus one for - // the leading 0 boundary); no growable, boxing list needed. - int[] boundaries = new int[target + 1]; - int chunkCount = 0; - Random rng = new Random(seed); - int outPos = 0; - - while (outPos < target) { - int row = nonEmptyRowIndices[rng.nextInt(nonEmptyRows)]; - int rowLen = access.rowLength(row); - int remaining = target - outPos; - int chunkLen = Math.min(Math.min(MAX_CHUNK_BYTES, rowLen), remaining); - int maxStart = rowLen - chunkLen; - int start = maxStart == 0 ? 0 : rng.nextInt(maxStart + 1); - access.copyChunk(row, start, chunkLen, out, outPos); - outPos += chunkLen; - chunkCount++; - boundaries[chunkCount] = outPos; + /// The first row from `start` on, wrapping around, that is not empty, or -1 when every row is. + private static int firstNonEmptyFrom(int start, int rowCount, RowAccess access) { + for (int i = start; i < rowCount; i++) { + if (access.rowLength(i) > 0) { + return i; + } } + for (int i = 0; i < start; i++) { + if (access.rowLength(i) > 0) { + return i; + } + } + return -1; + } - return new Sample(out, boundaries, chunkCount); + /// Rust's `fsst_hash`, the C++ reference's `FSST_HASH`. + /// + /// @param value the value to hash + /// @return its hash + static long hash(long value) { + return value * 2971215073L ^ value >>> 15; } - /// Row-length and chunk-copy operations abstracting over the two row storage shapes `draw` - /// accepts (one `byte[]` per row, or rows concatenated into a shared buffer). private interface RowAccess { int rowLength(int row); - void copyChunk(int row, int start, int len, byte[] out, int outPos); + void copy(int row, int start, int len, byte[] out, int outPos); } - /// Returns the concatenated sample bytes. Chunk boundaries in [#chunkStart(int)] / - /// [#chunkEnd(int)] delimit the parts that came from distinct rows. - /// - /// @return the sample bytes; never `null`, possibly empty byte[] bytes() { return bytes; } - /// Returns the number of chunks the sample is split into. Generation `gen` replays only the - /// first [#chunkCountForGeneration(int)] of these. - /// - /// @return the total chunk count - int chunkCount() { - return chunkCount; + int lineCount() { + return lineCount; } - /// Returns the start offset (inclusive) of chunk `i` within [#bytes()]. - /// - /// @param i the chunk index, in `0 .. chunkCount() - 1` - /// @return the chunk's start offset into the sample bytes - int chunkStart(int i) { - return chunkBoundaries[i]; + int lineStart(int i) { + return lineBoundaries[i]; } - /// Returns the end offset (exclusive) of chunk `i` within [#bytes()]. - /// - /// @param i the chunk index, in `0 .. chunkCount() - 1` - /// @return the chunk's end offset into the sample bytes - int chunkEnd(int i) { - return chunkBoundaries[i + 1]; + int lineEnd(int i) { + return lineBoundaries[i + 1]; } - /// Returns how many leading chunks generation `gen` (0-based) should replay, applying that - /// generation's growing sample fraction. Generation 0 sees roughly 6% of the chunks, the last - /// generation sees them all. At least one chunk is always replayed when the sample is non-empty, - /// so no generation trains on nothing. - /// - /// @param gen the 0-based generation index, in `0 .. SAMPLE_FRACTION_NUMERATORS.length - 1` - /// @return the number of leading chunks to replay for that generation - int chunkCountForGeneration(int gen) { - int total = chunkCount(); - if (total == 0) { - return 0; - } - long scaled = (long) total * SAMPLE_FRACTION_NUMERATORS[gen] / SAMPLE_FRACTION_DENOMINATOR; - return (int) Math.clamp(total, 0, scaled); + /// Whether the corpus was sampled down, rather than taken whole: Rust prunes the last + /// generation's table only for a corpus it took whole. + boolean sampled() { + return sampled; } } diff --git a/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java b/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java index 32fd88cf1..a4f6608db 100644 --- a/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java +++ b/fsst/src/main/java/io/github/dfa1/vortex/fsst/TrainingGeneration.java @@ -2,32 +2,27 @@ import java.util.ArrayList; import java.util.BitSet; -import java.util.Collections; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Map; -/// One generation of FSST bottom-up training (the FSST paper's Algorithm 3): compress-count the -/// sample with the current symbol table, then build the next table from the top-gain candidates. +/// One generation of FSST bottom-up training (the FSST paper's Algorithm 3), a port of Rust's +/// `fsst-rs` `Compressor::train` loop body (`builder.rs`): compress-count the generation's lines with +/// the current symbol table, then build the next table from the highest-gain candidates. /// -/// Compress-count parses each sample chunk with the current [Compressor] using longest-match-first -/// greedy parsing, tallying how often each matched symbol occurs (`count1`) and how often each pair -/// of adjacent matches occurs (`count2`). Make-table then proposes a candidate for every counted -/// symbol (its own bytes) and every counted adjacent pair (their concatenation, truncated to 8 -/// bytes), ranks them by gain, and keeps the top 255. +/// Compress-count parses each line with the current [Compressor], longest match first, tallying how +/// often each match occurs (`count1`) and each adjacent pair of matches (`count2`); a match of more +/// than one byte also counts its first byte alone, as the extension a single-byte symbol could +/// make. Counts are keyed by Rust's extended codes: a byte no symbol matches is its own value +/// `0..255`, and the symbol of code `k` is `256 + k`. /// -/// Beyond the bare paper this ports the Rust reference's four refinements (issue #287), each guarded -/// and documented at its site below: per-generation min-count pruning, the 8x single-byte gain -/// boost, and — on the final generation only — a cost-based prune pass. The growing sample fraction -/// (the fourth refinement) is applied by the caller via [Sample#chunkCountForGeneration(int)]. -/// -/// Counting is keyed by *code*, not by packed bytes (issue #393 #395): every match [Matcher] returns -/// already carries a code, and every non-match (escape) reduces to one of 256 literal byte values — -/// together a fixed, dense `[0, CODE_SPACE)` space matching the Rust reference's own -/// (`spiraldb/fsst/src/builder.rs`'s `Counter`, `FSST_CODE_MASK`-sized `count1`/`count2`). [Counts] -/// tallies both into flat arrays with no hashing, no boxing, and no per-generation reallocation — a -/// [BitSet] per array tracks which slots this generation actually touched, so clearing between -/// generations only resets the (small) touched-bit tracking, not the (large) count arrays -/// themselves. A single [Counts] instance is meant to be reused across every generation of one -/// [CompressorBuilder#train(byte[][])] call via [#run(Compressor, Sample, int, int, boolean, Counts)]. +/// Make-table turns every counted code into a candidate, the symbol itself and, until the last +/// generation, its merges with the codes seen after it, and sums the gains of candidates that are the +/// same symbol. The candidates are popped best first from a heap that orders them by gain and length +/// only, as Rust's `BinaryHeap` does, and the first 255 that fit become the table. Two candidates of +/// equal gain and length leave the heap in an order that depends on the order they were pushed, which +/// in Rust is its hash map's iteration order; here it is the order the codes were counted. A table +/// then differs from Rust's in at most such ties. final class TrainingGeneration { /// Maximum number of real symbols kept per generation (codes `0..254`; `0xFF` is the escape). @@ -36,219 +31,158 @@ final class TrainingGeneration { /// Maximum symbol length in bytes, bounded by what fits in one `long`. private static final int MAX_SYMBOL_LENGTH = 8; - /// First code in the escape pseudo-code range: real trained-symbol codes are always - /// `< ESCAPE_BASE` (bounded by [#MAX_SYMBOLS], since a [Matcher] built from `current` can never - /// return a code past its own table size), so `ESCAPE_BASE + byteValue` (`byteValue` in `0..255`) - /// gives every possible escaped literal byte its own code with no possibility of collision. - private static final int ESCAPE_BASE = MAX_SYMBOLS; - - /// Total dense code space: [#MAX_SYMBOLS] real-symbol slots plus 256 escape pseudo-codes, one - /// per literal byte value. Fixed regardless of how many symbols are actually trained so far in - /// any given generation, which is what lets [Counts]'s arrays stay a constant size and be reused - /// across generations instead of resized as the table grows. - private static final int CODE_SPACE = ESCAPE_BASE + 256; - - /// Single-byte gain multiplier. WHY: multiplying every length-1 candidate's gain by 8 before - /// ranking deliberately over-values frequent single bytes relative to their raw `count*length`. - /// A single byte in the table replaces a 2-byte escape on every occurrence, so keeping frequent - /// bytes suppresses escapes; without the boost a marginally-higher-gain multi-byte candidate can - /// crowd a genuinely more valuable single byte out of the 255 slots. The old encoder used plain - /// gain and had exactly this gap. - private static final int SINGLE_BYTE_GAIN_BOOST = 8; + /// Rust's `FSST_CODE_BASE`: the extended code of symbol 0. Codes below it are bytes no symbol + /// matched. + private static final int CODE_BASE = 256; - private TrainingGeneration() { - } + /// Rust's `FSST_CODE_MASK`: the extended code before any match in a line, never a real code. + private static final int NO_CODE = 511; - /// Runs one training generation with a fresh [Counts], for callers that do not need to reuse - /// counting storage across generations (tests; single-generation callers). - /// - /// @param current the current generation's compressor (an empty-table compressor for generation - /// 0, so every position escapes and bootstraps single-byte candidates) - /// @param sample the fixed training sample - /// @param chunkLimit the number of leading sample chunks this generation replays (its growing - /// fraction), from [Sample#chunkCountForGeneration(int)] - /// @param sampleFractionNumerator this generation's sample-fraction numerator over - /// [Sample#SAMPLE_FRACTION_DENOMINATOR], used for the min-count - /// floor - /// @param finalGeneration whether this is the last generation, which relaxes the min-count floor - /// to 1 and enables the cost-based final prune - /// @return the next generation's symbols, gain-descending, at most [#MAX_SYMBOLS] of them - static List run(Compressor current, Sample sample, int chunkLimit, - int sampleFractionNumerator, boolean finalGeneration) { - return run(current, sample, chunkLimit, sampleFractionNumerator, finalGeneration, new Counts()); - } + /// Size of the code space, Rust's `COUNTS1_SIZE`. + private static final int CODE_SPACE = 512; - /// Runs one training generation and returns the next generation's symbols in gain-descending - /// order (code = list index), ready to build the next [Compressor] / [Matcher]. - /// - /// @param current the current generation's compressor (an empty-table compressor for generation - /// 0, so every position escapes and bootstraps single-byte candidates) - /// @param sample the fixed training sample - /// @param chunkLimit the number of leading sample chunks this generation replays (its growing - /// fraction), from [Sample#chunkCountForGeneration(int)] - /// @param sampleFractionNumerator this generation's sample-fraction numerator over - /// [Sample#SAMPLE_FRACTION_DENOMINATOR], used for the min-count - /// floor - /// @param finalGeneration whether this is the last generation, which relaxes the min-count floor - /// to 1 and enables the cost-based final prune - /// @param counts reusable counting storage, cleared at the start of this call; passing the same - /// instance across every generation of one `train()` call avoids reallocating its - /// backing arrays five times over - /// @return the next generation's symbols, gain-descending, at most [#MAX_SYMBOLS] of them - static List run(Compressor current, Sample sample, int chunkLimit, - int sampleFractionNumerator, boolean finalGeneration, Counts counts) { - compressCount(current, sample, chunkLimit, counts); - return makeTable(current, counts, sampleFractionNumerator, finalGeneration); + /// Single-byte gain multiplier: a single byte in the table replaces a 2-byte escape on every + /// occurrence, so frequent bytes are worth more than their raw `count * length`. The C++ + /// reference's heuristic. + private static final int SINGLE_BYTE_GAIN_BOOST = 8; + + private TrainingGeneration() { } - /// Compress-counts the first `chunkLimit` chunks of `sample` into `counts` (cleared first): - /// greedily parses each chunk with `current`, tallying single-match counts and adjacent-pair - /// counts, both keyed by code (generation 0's empty table escapes everything, seeding the escape - /// pseudo-codes for single bytes). + /// Runs one training generation and returns the next generation's symbols in the order they + /// were taken (code = list index), ready to build the next [Compressor]. /// - /// Each position loads its 8-byte input word exactly once. While 8 real bytes remain, the word - /// loads with one intrinsified `VarHandle` read (the same trick [Compressor#compress(byte[], - /// int, int, byte[], long)] uses); only the tail of each chunk falls back to the byte-by-byte, - /// zero-padding [Compressor#loadWord(byte[], int, int)]. - private static void compressCount(Compressor current, Sample sample, int chunkLimit, Counts counts) { - counts.clear(); + /// @param current the current generation's compressor (an empty one for generation 0, so every + /// position escapes and bootstraps the single-byte candidates) + /// @param sample the training sample + /// @param fraction this generation's sample fraction numerator over + /// [Sample#FRACTION_DENOMINATOR]: it picks the lines replayed, the count floor, and + /// whether merges are made (not in the last generation) + /// @param counts reusable counting storage, cleared at the end of this call + /// @return the next generation's symbols, at most [#MAX_SYMBOLS] of them + static List run(Compressor current, Sample sample, int fraction, Counts counts) { byte[] bytes = sample.bytes(); - for (int c = 0; c < chunkLimit; c++) { - int start = sample.chunkStart(c); - int end = sample.chunkEnd(c); - int previousCode = -1; // No previous match within this chunk yet. - int pos = start; - int fastEnd = end - 8; - while (pos < end) { - long word = pos <= fastEnd - ? (long) Compressor.LONG_LE_BYTES.get(bytes, pos) - : Compressor.loadWord(bytes, pos, end); - int codeAndLength = codeAndLengthAt(current, word, pos, end); - int code = Matcher.codeOf(codeAndLength); - counts.bumpSingle(code); - if (previousCode >= 0) { - counts.bumpPair(previousCode, code); - } - previousCode = code; - pos += Matcher.lengthOf(codeAndLength); + for (int line = 0; line < sample.lineCount(); line++) { + // Rust's per-line skip: a generation replays about fraction/128 of the lines + if (fraction < Sample.FRACTION_DENOMINATOR && (Sample.hash(line) & 127) > fraction) { + continue; } + compressCount(current, bytes, sample.lineStart(line), sample.lineEnd(line), counts); } + // the last generation of a corpus taken whole is pruned, one that was sampled is not + boolean prune = fraction >= Sample.FRACTION_DENOMINATOR && !sample.sampled(); + List next = optimize(current, counts, fraction, prune); + counts.clear(); + return next; } - /// Returns the code and length of the current table's longest match against the already-loaded - /// `word` at `pos`, packed as `code << 8 | length` (the same shape [Matcher#longestMatch(long)] - /// itself returns). Falls back to the escape pseudo-code for the literal byte at `pos` (length 1) - /// when nothing matches. - /// - /// An over-long match is rejected the same way [Compressor#compress(byte[], int, int, byte[], - /// long)] rejects it: `word` zero-pads past `end` when loaded via [Compressor#loadWord(byte[], - /// int, int)], so a symbol with trailing zero bytes can spuriously match the padding beyond the - /// chunk. Counting such a match would tally a candidate spanning bytes that are not really in - /// the sample, and could advance `pos` past `end`. - private static int codeAndLengthAt(Compressor current, long word, int pos, int end) { - int packedMatch = current.matcher().longestMatch(word); - int length = Matcher.lengthOf(packedMatch); - if (length > 0 && pos + length <= end) { - return packedMatch; + private static void compressCount(Compressor current, byte[] bytes, int start, int end, Counts counts) { + int previousCode = NO_CODE; + int pos = start; + int fastEnd = end - 8; + while (pos < end) { + long word = pos <= fastEnd + ? (long) Compressor.LONG_LE_BYTES.get(bytes, pos) + : Compressor.loadWord(bytes, pos, end); + int packedMatch = current.matcher().longestMatch(word); + int length = Matcher.lengthOf(packedMatch); + int code; + if (length > 0) { + code = CODE_BASE + Matcher.codeOf(packedMatch); + } else { + code = (int) (word & 0xFF); + length = 1; + } + counts.bumpSingle(code); + counts.bumpPair(previousCode, code); + if (length > 1) { + // also the extension by just this symbol's first byte + int firstByte = (int) (word & 0xFF); + counts.bumpSingle(firstByte); + counts.bumpPair(previousCode, firstByte); + } + previousCode = code; + pos += length; } - return (ESCAPE_BASE + (int) (word & 0xFF)) << 8 | 1; } - /// Builds the next table from the counted candidates: proposes each single symbol and each - /// adjacent-pair concatenation, prunes by min-count, boosts single-byte gain, ranks, applies the - /// final cost prune on the last generation, and keeps the top [#MAX_SYMBOLS] by gain. - /// - /// @param current the compressor `counts` was tallied against — real (non-escape) codes resolve - /// to their bytes/length through it - private static List makeTable(Compressor current, Counts counts, - int sampleFractionNumerator, boolean finalGeneration) { - // Refinement 2 — per-generation min-count pruning. A candidate seen fewer times than this - // floor is noise at this generation's sample coverage and is dropped before its gain is even - // computed. The floor scales with the sample fraction so early, low-coverage generations - // prune more aggressively; the final generation relaxes it to 1 so rare-but-real candidates - // survive the last, authoritative ranking pass. - int minCount = finalGeneration - ? 1 - : Math.max(1, 5 * sampleFractionNumerator / Sample.SAMPLE_FRACTION_DENOMINATOR); - - List candidates = new ArrayList<>(); - counts.forEachSingle((code, count) -> { - if (count >= minCount) { - int length = candidateLength(current, code); - long packed = candidateBytes(current, code); - candidates.add(new Candidate(packed, length, gainOf(count, length))); - } - }); - counts.forEachPair((firstCode, secondCode, count) -> { + /// Rust's `optimize`: the candidates from the counts, then the best 255 that fit in the table. + private static List optimize(Compressor current, Counts counts, int fraction, boolean prune) { + // keyed by the symbol's bytes: the same symbol reached through different codes (a byte + // that escaped and the symbol that is just that byte) adds up its gains + Map gains = new LinkedHashMap<>(); + // Rust's integer division: 0 at the first generation, where every count is at least 1 + int minCount = prune ? 1 : 5 * fraction / Sample.FRACTION_DENOMINATOR; + for (int code1 = counts.nextSingle(0); code1 >= 0 && code1 < NO_CODE; code1 = counts.nextSingle(code1 + 1)) { + long symbol1 = symbolOf(current, code1); + int length1 = lengthOf(symbol1); + long count = counts.single(code1); if (count < minCount) { - return; + continue; } - int firstLength = candidateLength(current, firstCode); - long firstPacked = candidateBytes(current, firstCode); - int secondLength = candidateLength(current, secondCode); - long secondPacked = candidateBytes(current, secondCode); - int combinedLength = Math.min(firstLength + secondLength, MAX_SYMBOL_LENGTH); - long combined = concatenate(firstPacked, firstLength, secondPacked); - long packed = combined & lengthMask(combinedLength); - candidates.add(new Candidate(packed, combinedLength, gainOf(count, combinedLength))); - }); - - return selectTop(candidates, finalGeneration); - } - - /// Resolves a code's packed bytes: `current`'s trained symbol table for a real code, or the - /// literal byte value itself for an escape pseudo-code. - private static long candidateBytes(Compressor current, int code) { - return code < ESCAPE_BASE ? current.packedSymbol(code) : (long) (code - ESCAPE_BASE); - } - - /// Resolves a code's byte length: `current`'s trained symbol table for a real code, or 1 for an - /// escape pseudo-code (every escape is a single literal byte). - private static int candidateLength(Compressor current, int code) { - return code < ESCAPE_BASE ? current.symbolLength(code) : 1; - } - - /// Ranks candidates by gain-descending (length-descending on ties) and keeps the top - /// [#MAX_SYMBOLS], applying the final cost prune on the last generation and skipping a 3-8 byte - /// candidate whose [LossyPerfectHashTable] slot a better one already holds. - private static List selectTop(List candidates, boolean finalGeneration) { - // Refinement 4 — final cost-based prune, applied BEFORE top-K selection. Only on the last - // generation, a symbol earns its code slot only if its real (un-boosted) gain exceeds - // length+1: it must save at least that many bytes over escaping every occurrence, otherwise - // the slot is better left for a candidate that does. Pruning after the heap (as opposed to - // before) would leave a disqualified survivor's slot empty even when a qualifying candidate - // sat just below the top-255 cut, so the 255 slots would not all be filled with symbols that - // earn them. Provisional earlier generations skip this so their symbols stay available to be - // re-evaluated next pass. - List eligible = candidates; - if (finalGeneration) { - eligible = new ArrayList<>(candidates.size()); - for (Candidate candidate : candidates) { - if (realGain(candidate) > candidate.length() + 1L) { - eligible.add(candidate); + long gain = count * length1; + if (length1 == 1) { + gain *= SINGLE_BYTE_GAIN_BOOST; + } + gains.merge(symbol1, gain, Long::sum); + // no merges in the last generation, nor of a symbol that cannot grow + if (fraction >= Sample.FRACTION_DENOMINATOR || length1 == MAX_SYMBOL_LENGTH) { + continue; + } + for (int code2 = counts.nextPair(code1, 0); code2 >= 0 && code2 < NO_CODE; + code2 = counts.nextPair(code1, code2 + 1)) { + long symbol2 = symbolOf(current, code2); + int length2 = lengthOf(symbol2); + if (length1 + length2 > MAX_SYMBOL_LENGTH) { + continue; } + long merged = symbol1 | symbol2 << (8 * length1); + gains.merge(merged, counts.pair(code1, code2) * lengthOf(merged), Long::sum); } } - // Best first, as Rust's `optimize` pops its heap: a 3-8 byte candidate whose hash slot a - // better one holds is skipped, and the next candidate takes its code. Rust asserts that no - // two stored symbols share a slot when it rebuilds a compressor from a file's symbol table - // (vortex-jni does, to compress a filter literal), so a table breaking it aborted the JVM. - eligible.sort(Collections.reverseOrder(TrainingGeneration::compareGain)); + CandidateHeap heap = new CandidateHeap(gains.size()); + for (Map.Entry candidate : gains.entrySet()) { + heap.push(candidate.getValue(), lengthOf(candidate.getKey()), candidate.getKey()); + } BitSet takenSlots = new BitSet(); List result = new ArrayList<>(MAX_SYMBOLS); - for (int i = 0; i < eligible.size() && result.size() < MAX_SYMBOLS; i++) { - Candidate candidate = eligible.get(i); - if (candidate.length() < 3 || claimSlot(takenSlots, candidate.packed())) { - result.add(new Symbol(candidate.packed(), candidate.length())); + while (!heap.isEmpty() && result.size() < MAX_SYMBOLS) { + heap.pop(); + long symbol = heap.poppedSymbol(); + int length = heap.poppedLength(); + if (prune) { + long saves = heap.poppedGain(); + if (length == 1) { + saves /= SINGLE_BYTE_GAIN_BOOST; + } + if (saves <= length + 1) { + continue; + } + } + // a 3-8 byte symbol whose hash slot a better one holds is skipped, and the next + // candidate takes its code + if (length < 3 || claimSlot(takenSlots, symbol)) { + result.add(new Symbol(symbol, length)); } } return result; } - /// Takes the [LossyPerfectHashTable] slot of a 3-8 byte symbol. - /// - /// @return `false` when a better symbol already holds it + /// The symbol of an extended code: a byte no symbol matched, or the current table's symbol. + private static long symbolOf(Compressor current, int code) { + if (code < CODE_BASE) { + return code; + } + return current.packedSymbol(code - CODE_BASE); + } + + /// Rust's `Symbol::len`: the bytes up to the last non-zero one, at least one. A symbol that ends in + /// zero bytes is the shorter one, and a zero byte alone is one byte. + private static int lengthOf(long symbol) { + return Math.max(1, MAX_SYMBOL_LENGTH - Long.numberOfLeadingZeros(symbol) / 8); + } + private static boolean claimSlot(BitSet takenSlots, long packed) { int slot = LossyPerfectHashTable.slotFor(packed); if (takenSlots.get(slot)) { @@ -258,62 +192,133 @@ private static boolean claimSlot(BitSet takenSlots, long packed) { return true; } - /// Gain used for ranking: `count * length`, with the single-byte boost applied to length-1 - /// candidates (refinement 3). - private static long gainOf(long count, int length) { - long gain = count * length; - return length == 1 ? gain * SINGLE_BYTE_GAIN_BOOST : gain; - } + /// Rust's `BinaryHeap`, sift for sift, ordered by `(gain, length)`: the order equal + /// candidates pop in is the heap's own, so it has to be the same algorithm. + static final class CandidateHeap { + + private final long[] gains; + private final int[] lengths; + private final long[] symbols; + private int size; + private long poppedGain; + private int poppedLength; + private long poppedSymbol; + + CandidateHeap(int capacity) { + gains = new long[capacity]; + lengths = new int[capacity]; + symbols = new long[capacity]; + } - /// Real (un-boosted) gain of a candidate, dividing the length-1 boost back out — used only by - /// the final cost prune, which must judge a symbol on its true byte savings, not its boosted - /// ranking gain. - private static long realGain(Candidate candidate) { - return candidate.length() == 1 - ? candidate.gain() / SINGLE_BYTE_GAIN_BOOST - : candidate.gain(); - } + boolean isEmpty() { + return size == 0; + } - private static int compareGain(Candidate a, Candidate b) { - int byGain = Long.compare(a.gain(), b.gain()); - if (byGain != 0) { - return byGain; + long poppedGain() { + return poppedGain; } - return Integer.compare(a.length(), b.length()); - } - /// Concatenates two packed symbols LSB-first: the first symbol's bytes stay in the low bytes and - /// the second symbol's bytes are shifted up by the first symbol's byte length, so the combined - /// word reads the first symbol's bytes then the second's. A shift of 64 bits (first already 8 - /// bytes long) drops the second symbol entirely, which is correct — the result is then truncated - /// to 8 bytes anyway. - private static long concatenate(long first, int firstLength, long second) { - if (firstLength >= MAX_SYMBOL_LENGTH) { - return first; + int poppedLength() { + return poppedLength; } - return first | (second << (firstLength * 8)); - } - private static long lengthMask(int length) { - return length >= MAX_SYMBOL_LENGTH ? ~0L : (1L << (length * 8)) - 1; - } + long poppedSymbol() { + return poppedSymbol; + } + + void push(long gain, int length, long symbol) { + gains[size] = gain; + lengths[size] = length; + symbols[size] = symbol; + siftUp(0, size); + size++; + } + + /// Rust's `pop`: takes the last element, swaps it with the root, and sifts the root down to + /// the bottom and back up. + void pop() { + size--; + long gain = gains[size]; + int length = lengths[size]; + long symbol = symbols[size]; + if (size > 0) { + long rootGain = gains[0]; + int rootLength = lengths[0]; + long rootSymbol = symbols[0]; + gains[0] = gain; + lengths[0] = length; + symbols[0] = symbol; + gain = rootGain; + length = rootLength; + symbol = rootSymbol; + siftDownToBottom(0); + } + poppedGain = gain; + poppedLength = length; + poppedSymbol = symbol; + } + + private boolean lessOrEqual(long gainA, int lengthA, long gainB, int lengthB) { + if (gainA != gainB) { + return gainA < gainB; + } + return lengthA <= lengthB; + } + + private void siftUp(int start, int position) { + long gain = gains[position]; + int length = lengths[position]; + long symbol = symbols[position]; + int pos = position; + while (pos > start) { + int parent = (pos - 1) / 2; + if (lessOrEqual(gain, length, gains[parent], lengths[parent])) { + break; + } + gains[pos] = gains[parent]; + lengths[pos] = lengths[parent]; + symbols[pos] = symbols[parent]; + pos = parent; + } + gains[pos] = gain; + lengths[pos] = length; + symbols[pos] = symbol; + } - /// A ranked candidate symbol: packed bytes, length, and ranking gain (boosted for length 1). - private record Candidate(long packed, int length, long gain) { + private void siftDownToBottom(int position) { + int end = size; + int start = position; + long gain = gains[position]; + int length = lengths[position]; + long symbol = symbols[position]; + int pos = position; + int child = 2 * pos + 1; + while (child <= Math.max(0, end - 2)) { + if (lessOrEqual(gains[child], lengths[child], gains[child + 1], lengths[child + 1])) { + child++; + } + gains[pos] = gains[child]; + lengths[pos] = lengths[child]; + symbols[pos] = symbols[child]; + pos = child; + child = 2 * pos + 1; + } + if (child == end - 1) { + gains[pos] = gains[child]; + lengths[pos] = lengths[child]; + symbols[pos] = symbols[child]; + pos = child; + } + gains[pos] = gain; + lengths[pos] = length; + symbols[pos] = symbol; + siftUp(start, pos); + } } - /// Tallies of single-match and adjacent-pair counts during compress-count, keyed by *code* — a - /// dense `[0, CODE_SPACE)` space, not packed bytes — matching the Rust reference's own - /// `Counter` (`spiraldb/fsst/src/builder.rs`, issue #393 #395). `counts1`/`counts2` are flat - /// arrays sized once and reused across every generation of a `train()` call: a code's identity - /// alone (not its byte content) is the key, so there is no hashing, no boxing, and — since the - /// code space is fixed regardless of how many symbols are actually trained in any one - /// generation — no resizing either. - /// - /// Between generations only [#clear()] runs, which resets the touched-bit tracking, not the - /// (much larger) count arrays: [#bumpSingle(int)]/[#bumpPair(int, int)] read a slot's prior value - /// only when its touched bit is set, so a stale value left over from an earlier generation in an - /// untouched slot is never read. + /// Match counts and adjacent-pair counts over the extended code space, flat arrays with no + /// hashing, reused across generations: a [BitSet] per array tracks the slots a generation + /// touched, so clearing resets only that. static final class Counts { private final long[] counts1 = new long[CODE_SPACE]; @@ -321,9 +326,6 @@ static final class Counts { private final BitSet singleTouched = new BitSet(CODE_SPACE); private final BitSet pairTouched = new BitSet(CODE_SPACE * CODE_SPACE); - /// Resets counting for a new generation. Only the touched-bit tracking is cleared — cheap, - /// proportional to how much was actually touched last generation — not the count arrays - /// themselves. void clear() { singleTouched.clear(); pairTouched.clear(); @@ -342,26 +344,27 @@ void bumpPair(int firstCode, int secondCode) { pairTouched.set(idx); } - void forEachSingle(SingleConsumer consumer) { - for (int code = singleTouched.nextSetBit(0); code >= 0; code = singleTouched.nextSetBit(code + 1)) { - consumer.accept(code, counts1[code]); - } + long single(int code) { + return counts1[code]; } - void forEachPair(PairConsumer consumer) { - for (int idx = pairTouched.nextSetBit(0); idx >= 0; idx = pairTouched.nextSetBit(idx + 1)) { - consumer.accept(idx / CODE_SPACE, idx % CODE_SPACE, counts2[idx]); - } + long pair(int firstCode, int secondCode) { + return counts2[firstCode * CODE_SPACE + secondCode]; } - } - @FunctionalInterface - private interface SingleConsumer { - void accept(int code, long count); - } + /// The first counted code at or after `from`, in ascending order, or -1. + int nextSingle(int from) { + return singleTouched.nextSetBit(from); + } - @FunctionalInterface - private interface PairConsumer { - void accept(int firstCode, int secondCode, long count); + /// The first code at or after `from` counted as following `firstCode`, in ascending order, or + /// -1. + int nextPair(int firstCode, int from) { + int idx = pairTouched.nextSetBit(firstCode * CODE_SPACE + from); + if (idx < 0 || idx >= (firstCode + 1) * CODE_SPACE) { + return -1; + } + return idx - firstCode * CODE_SPACE; + } } } diff --git a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateHeapTest.java b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateHeapTest.java new file mode 100644 index 000000000..9bb5cde12 --- /dev/null +++ b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CandidateHeapTest.java @@ -0,0 +1,90 @@ +package io.github.dfa1.vortex.fsst; + +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.Random; + +import static org.assertj.core.api.Assertions.assertThat; + +/// The heap equal candidates pop out of, as Rust's `BinaryHeap` pops them. +class CandidateHeapTest { + + @Test + void pop_candidatesOfEqualGain_leaveInRustsHeapOrderNotPushOrder() { + // Given A5, B3, C5, D9 pushed in that order. By Rust's push (sift_up stops at an equal + // parent) and pop (swap in the last, sift down to the bottom, sift back up): D, C, A, B + TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(4); + sut.push(5, 1, 'A'); + sut.push(3, 1, 'B'); + sut.push(5, 1, 'C'); + sut.push(9, 1, 'D'); + + // When + List popped = drain(sut); + + // Then + assertThat(popped).containsExactly((long) 'D', (long) 'C', (long) 'A', (long) 'B'); + } + + @Test + void pop_allEqual_leavesInPushOrder() { + // Given three candidates of the same gain and length + TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(3); + sut.push(1, 1, 'A'); + sut.push(1, 1, 'B'); + sut.push(1, 1, 'C'); + + // When + List popped = drain(sut); + + // Then + assertThat(popped).containsExactly((long) 'A', (long) 'B', (long) 'C'); + } + + @Test + void pop_orderedByGainThenLength() { + // Given a longer symbol and a shorter one of the same gain + TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(3); + sut.push(7, 2, 'S'); + sut.push(7, 5, 'L'); + sut.push(9, 1, 'G'); + + // When + List popped = drain(sut); + + // Then the higher gain first, then the longer symbol + assertThat(popped).containsExactly((long) 'G', (long) 'L', (long) 'S'); + } + + @Test + void pop_randomCandidates_yieldsEachOnceInNonIncreasingOrder() { + // Given 500 candidates with many equal gains + Random random = new Random(11); + TrainingGeneration.CandidateHeap sut = new TrainingGeneration.CandidateHeap(500); + long[] gains = new long[500]; + for (int i = 0; i < gains.length; i++) { + gains[i] = random.nextInt(40); + sut.push(gains[i], 1 + random.nextInt(8), i); + } + + // When + List popped = drain(sut); + + // Then + assertThat(popped).hasSize(500).doesNotHaveDuplicates(); + for (int i = 1; i < popped.size(); i++) { + assertThat(gains[(int) (long) popped.get(i)]).isLessThanOrEqualTo(gains[(int) (long) popped.get(i - 1)]); + } + } + + private static List drain(TrainingGeneration.CandidateHeap heap) { + List popped = new ArrayList<>(); + while (!heap.isEmpty()) { + heap.pop(); + popped.add(heap.poppedSymbol()); + } + return popped; + } +} diff --git a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorBuilderTest.java b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorBuilderTest.java index 685e79755..81eb0d3f1 100644 --- a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorBuilderTest.java +++ b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorBuilderTest.java @@ -6,6 +6,7 @@ import org.junit.jupiter.params.provider.ValueSource; import java.nio.charset.StandardCharsets; +import java.util.Arrays; import java.util.List; import java.util.Random; @@ -130,13 +131,13 @@ void compress_symbolWithTrailingZeros_doesNotMatchPastEndOfInput() { class Determinism { @Test - void train_sameInputAndSeed_producesByteIdenticalTable() { - // Given — the same corpus trained twice with the same explicit seed. + void train_sameInput_producesByteIdenticalTable() { + // Given — the same corpus trained twice. byte[][] rows = logLikeRows(2_000); // When - Compressor first = new CompressorBuilder().seed(1234L).train(rows); - Compressor second = new CompressorBuilder().seed(1234L).train(rows); + Compressor first = new CompressorBuilder().train(rows); + Compressor second = new CompressorBuilder().train(rows); // Then — identical symbol count and identical (packed, length) for every code. assertThat(second.symbolCount()).isEqualTo(first.symbolCount()); @@ -146,21 +147,6 @@ void train_sameInputAndSeed_producesByteIdenticalTable() { } } - @Test - void train_defaultSeed_isAlsoDeterministic() { - // Given — no explicit seed: the default seed must be a fixed constant, not wall-clock. - byte[][] rows = logLikeRows(1_000); - - // When - Compressor first = new CompressorBuilder().train(rows); - Compressor second = new CompressorBuilder().train(rows); - - // Then - assertThat(second.symbolCount()).isEqualTo(first.symbolCount()); - for (int code = 0; code < first.symbolCount(); code++) { - assertThat(second.packedSymbol(code)).isEqualTo(first.packedSymbol(code)); - } - } } @Nested @@ -202,79 +188,68 @@ class MinCountPruning { @Test void run_nonFinalGeneration_prunesCandidateBelowFloor() { // Given — "ab" repeated a thousand times (very frequent) with a single "cd" appended - // (count 1). A non-final generation at full sample fraction (numerator 128) uses a - // min-count floor of 5, so the rare "cd" pair — and its bytes 'c'/'d' — must be pruned + // (count 1). A non-final generation scales its min-count floor with its sample + // fraction (5 * 98 / 128 = 3 here), so the rare "cd" pair — and its bytes 'c'/'d' — must be pruned // before ranking, unlike a naive keep-everything scheme. StringBuilder text = new StringBuilder(); for (int i = 0; i < 1_000; i++) { text.append("ab"); } text.append("cd"); - Sample sample = Sample.draw(new byte[][]{utf8(text.toString())}, 1L); + Sample sample = Sample.draw(new byte[][]{utf8(text.toString())}); Compressor emptyTable = Compressor.of(List.of()); - // When — one non-final generation over the whole sample. + // When — one non-final generation (fraction 98 of 128: a floor of 5 * 98 / 128 = 3). List result = - TrainingGeneration.run(emptyTable, sample, sample.chunkCount(), 128, false); + TrainingGeneration.run(emptyTable, sample, 98, new TrainingGeneration.Counts()); - // Then — the rare bytes never make it into the table. - assertThat(containsByte(result, (byte) 'c')).isFalse(); - assertThat(containsByte(result, (byte) 'd')).isFalse(); + // Then — the rare bytes are no symbols of their own (a merge with a frequent code may carry + // them, as Rust's candidates do: its floor is on the first code's count) + assertThat(hasSymbol(result, 1, (byte) 'c')).isFalse(); + assertThat(hasSymbol(result, 1, (byte) 'd')).isFalse(); // Sanity: the frequent 'a'/'b' content is retained. assertThat(containsByte(result, (byte) 'a')).isTrue(); } } @Nested - class SingleByteGainBoost { + class Growth { @Test - void train_frequentSingleByte_survivesAmidManyMultiByteCompetitors() { - // Given — a frequent single byte 'x' separates many distinct 3-byte groups. Under plain - // count*length gain a single byte is easily out-ranked by longer candidates; the 8x - // single-byte boost is exactly what keeps a frequent byte in the table, so it is not - // crowded out and its occurrences do not degrade into 2-byte escapes. - String[] groups = {"abc", "def", "ghi", "jkl", "mno", "pqr", "stu", "vwx"}; - Random random = new Random(3); - StringBuilder text = new StringBuilder(); - for (int i = 0; i < 4_000; i++) { - text.append(groups[random.nextInt(groups.length)]).append('x'); - } - Compressor sut = new CompressorBuilder().train(new byte[][]{utf8(text.toString())}); + void train_aRunOfOneByte_growsEightByteSymbols() { + // Given — rows of one repeated byte: each generation doubles the symbol by merging a + // symbol with itself, up to the eight bytes a symbol holds + byte[] row = new byte[1_000]; + Arrays.fill(row, (byte) 'x'); // When - boolean result = hasSingleByteSymbol(sut, (byte) 'x'); + Compressor sut = new CompressorBuilder().train(new byte[][]{row, row, row}); // Then - assertThat(result).isTrue(); + assertThat(hasSymbol(sut, 8, (byte) 'x', (byte) 'x', (byte) 'x', (byte) 'x', + (byte) 'x', (byte) 'x', (byte) 'x', (byte) 'x')).isTrue(); } } @Nested - class NulByteCollision { + class NulByteSymbols { @Test - void run_currentTableAlreadyHasNulSuffixedSymbol_countsBothMatchLengthsSeparately() { - // Given — a current table that already contains "A\0" as a trained 2-byte symbol (as - // it would after an earlier generation learned it from real "A\0" occurrences). The - // sample repeats "AX" (a standalone 'A' the matcher can't extend into "A\0", since the - // next byte is 'X' not NUL) and "A\0" (which the matcher matches as one length-2 - // symbol) equally often. Both produce a `bumpSingle` call with the *same* packed long: - // `{0x41}` (length 1, from "AX"'s 'A') and `{0x41, 0x00}` (length 2, from "A\0") both - // mask to `0x41` (issue #393 #5). A packed-only key would merge the two into one - // corrupted candidate; keying by `(packed, length)` must keep them distinct. - Compressor current = Compressor.of(List.of(new Symbol(0x41L, 2))); // "A\0", code 0 + void train_dataWithZeroBytes_neverTakesASymbolEndingInOne() { + // Given — "A" followed by a zero byte, and "AX", equally often. Rust measures a symbol by + // its last non-zero byte, so the merge of "A" and a zero byte is just "A": a symbol + // ending in zeros would have a length its bytes do not show. byte[] row = interleavedStandaloneAndNulSuffixed(2_000); - Sample sample = Sample.draw(new byte[][]{row}, 1L); - // When — one final generation (minCount relaxed to 1) over the whole sample. - List result = - TrainingGeneration.run(current, sample, sample.chunkCount(), 128, true); + // When + Compressor sut = new CompressorBuilder().train(new byte[][]{row}); - // Then — both the standalone length-1 'A' symbol and the real "A\0" length-2 symbol - // survive as distinct, correctly counted candidates. - assertThat(hasSymbol(result, 1, (byte) 'A')).isTrue(); - assertThat(hasSymbol(result, 2, (byte) 'A', (byte) 0)).isTrue(); + // Then — every symbol's length is where its last non-zero byte is + for (int code = 0; code < sut.symbolCount(); code++) { + long packed = sut.packedSymbol(code); + int length = Math.max(1, 8 - Long.numberOfLeadingZeros(packed) / 8); + assertThat(sut.symbolLength(code)).as("symbol %d", code).isEqualTo(length); + } } private byte[] interleavedStandaloneAndNulSuffixed(int repeats) { @@ -282,31 +257,12 @@ private byte[] interleavedStandaloneAndNulSuffixed(int repeats) { int pos = 0; for (int i = 0; i < repeats; i++) { row[pos++] = 'A'; - row[pos++] = 'X'; // standalone 'A': the matcher cannot extend this into "A\0" + row[pos++] = 'X'; row[pos++] = 'A'; - row[pos++] = 0; // "A\0": matches the pre-trained length-2 symbol + row[pos++] = 0; } return row; } - - private boolean hasSymbol(List symbols, int length, byte... expectedBytes) { - for (Symbol symbol : symbols) { - if (symbol.length() != length) { - continue; - } - boolean matches = true; - for (int k = 0; k < length; k++) { - if (symbol.byteAt(k) != expectedBytes[k]) { - matches = false; - break; - } - } - if (matches) { - return true; - } - } - return false; - } } private static byte[][] logLikeRows(int count) { @@ -351,8 +307,8 @@ void train_contiguousRows_matchesPerRowTableExactly() { } // When - Compressor perRow = new CompressorBuilder().seed(99L).train(rows); - Compressor result = new CompressorBuilder().seed(99L) + Compressor perRow = new CompressorBuilder().train(rows); + Compressor result = new CompressorBuilder() .train(flatten(rows), offsetsOf(rows), rows.length); // Then @@ -409,15 +365,34 @@ private int[] offsetsOf(byte[][] rows) { } } - private static boolean hasSingleByteSymbol(Compressor compressor, byte value) { + private static boolean hasSymbol(List symbols, int length, byte... bytes) { + for (Symbol symbol : symbols) { + if (symbol.length() == length && matches(symbol, bytes)) { + return true; + } + } + return false; + } + + private static boolean hasSymbol(Compressor compressor, int length, byte... bytes) { for (int code = 0; code < compressor.symbolCount(); code++) { - if (compressor.symbolLength(code) == 1 && (byte) compressor.packedSymbol(code) == value) { + if (compressor.symbolLength(code) == length + && matches(new Symbol(compressor.packedSymbol(code), length), bytes)) { return true; } } return false; } + private static boolean matches(Symbol symbol, byte[] bytes) { + for (int k = 0; k < bytes.length; k++) { + if (symbol.byteAt(k) != bytes[k]) { + return false; + } + } + return true; + } + private static boolean containsByte(List symbols, byte value) { for (Symbol symbol : symbols) { for (int k = 0; k < symbol.length(); k++) { diff --git a/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorWireOrderTest.java b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorWireOrderTest.java new file mode 100644 index 000000000..fbce1be6a --- /dev/null +++ b/fsst/src/test/java/io/github/dfa1/vortex/fsst/CompressorWireOrderTest.java @@ -0,0 +1,71 @@ +package io.github.dfa1.vortex.fsst; + +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/// The order Rust's `fsst-rs` numbers a final table in (`finalize`): 2-byte symbols no longer symbol +/// starts with, then the 2-byte symbols one does (last taken first), then 3 to 8 bytes by length in the +/// order taken, then the single bytes. +class CompressorWireOrderTest { + + @Test + void codesSortedByLength_followsRustsFinalize() { + // Given symbols taken in this order: code 0 is "q" ... code 7 is "yz" + Compressor sut = Compressor.of(List.of( + symbol("q"), + symbol("ab"), + symbol("abc"), + symbol("xy"), + symbol("wxyz"), + symbol("ab" + "d"), + symbol("r"), + symbol("yz"))); + + // When + int[] order = sut.codesSortedByLength(); + + // Then 2-byte symbols nothing extends ("xy", "yz") first in the order taken; the extended + // "ab" next; 3 bytes ("abc", "abd") then 4 ("wxyz"); single bytes ("q", "r") last + assertThat(order).containsExactly(3, 7, 1, 2, 5, 4, 0, 6); + } + + @Test + void codesSortedByLength_twoByteSymbolsSharingAPrefixWithALongerOneComeLastTakenFirst() { + // Given two 2-byte symbols each a prefix of a longer one, taken "ab" first + Compressor sut = Compressor.of(List.of(symbol("ab"), symbol("cd"), symbol("abx"), symbol("cdx"))); + + // When + int[] order = sut.codesSortedByLength(); + + // Then the later one ("cd") sits before the earlier ("ab"), then the 3-byte ones + assertThat(order).containsExactly(1, 0, 2, 3); + } + + @Test + void codesSortedByLength_asksEverySlotOfRustsTable_theOnesAnEarlierGenerationLeftToo() { + // Given a table of "ab", "cd" and "abx", and a later, smaller one of "ab" and "cd": Rust never + // clears the slot past the new symbols, so "abx" is still there to start with "ab" + Compressor earlier = Compressor.of(List.of(symbol("ab"), symbol("cd"), symbol("abx"))); + Compressor sut = Compressor.rebuild(List.of(symbol("ab"), symbol("cd")), earlier); + + // When + int[] order = sut.codesSortedByLength(); + + // Then "cd", which nothing starts with, before "ab" + assertThat(order).containsExactly(1, 0); + } + + @Test + void codesSortedByLength_emptyTable_isEmpty() { + // Given / When / Then + assertThat(Compressor.of(List.of()).codesSortedByLength()).isEmpty(); + } + + private static Symbol symbol(String text) { + return Symbol.of(text.getBytes(StandardCharsets.US_ASCII), 0, text.length()); + } +} diff --git a/fsst/src/test/java/io/github/dfa1/vortex/fsst/SampleTest.java b/fsst/src/test/java/io/github/dfa1/vortex/fsst/SampleTest.java index e80825925..5deff5abc 100644 --- a/fsst/src/test/java/io/github/dfa1/vortex/fsst/SampleTest.java +++ b/fsst/src/test/java/io/github/dfa1/vortex/fsst/SampleTest.java @@ -9,78 +9,118 @@ class SampleTest { @Test - void draw_largeInput_totalSampledBytesNearTarget() { - // Given — a corpus far larger than the 16KB target, so sampling must cap the total. + void draw_largeInput_totalSampledBytesPassTheTargetByAtMostALine() { + // Given a corpus far larger than the 16KB target, so it is sampled down byte[][] rows = repeatedRows("the quick brown fox jumps over the lazy dog ", 5_000); // When - Sample result = Sample.draw(rows, 1L); + Sample result = Sample.draw(rows); - // Then — the total lands at the target exactly here, since every draw chunk is full-length; - // in general it is bounded by TARGET_SAMPLE_BYTES and never overshoots (draws stop once the - // target is reached). Allow a small tolerance for the last, possibly-truncated chunk. + // Then Rust draws lines until the target is reached, so the last one may pass it + assertThat(result.sampled()).isTrue(); + assertThat(result.bytes().length).isGreaterThanOrEqualTo(Sample.TARGET_SAMPLE_BYTES); assertThat(totalBytes(result)) - .isLessThanOrEqualTo(Sample.TARGET_SAMPLE_BYTES) - .isGreaterThan(Sample.TARGET_SAMPLE_BYTES - Sample.MAX_CHUNK_BYTES); + .isGreaterThanOrEqualTo(Sample.TARGET_SAMPLE_BYTES) + .isLessThan(Sample.TARGET_SAMPLE_BYTES + Sample.LINE_BYTES); } @Test - void draw_smallInput_usesFullCorpus() { - // Given — a corpus well under the target: no under-sampling should occur. - byte[][] rows = repeatedRows("abc", 4); // 12 bytes total, << 16KB. + void draw_smallInput_keepsEveryRowWholeAsALine() { + // Given a corpus under the target, with an empty row among them + byte[][] rows = {bytes("abc"), new byte[0], bytes("de"), bytes("fghij")}; // When - Sample result = Sample.draw(rows, 1L); + Sample result = Sample.draw(rows); + + // Then every row is a line, the empty one too: training skips lines by their index + assertThat(result.sampled()).isFalse(); + assertThat(result.lineCount()).isEqualTo(4); + assertThat(result.lineEnd(1) - result.lineStart(1)).isZero(); + assertThat(totalBytes(result)).isEqualTo(10); + assertThat(new String(result.bytes(), StandardCharsets.UTF_8)).isEqualTo("abcdefghij"); + } + + @Test + void draw_sampledCorpusOfMostlyEmptyRows_walksPastTheEmptyOnes() { + // Given 10,000 rows of which every hundredth holds 400 bytes: 40KB, sampled + byte[][] rows = new byte[10_000][]; + for (int i = 0; i < rows.length; i++) { + rows[i] = i % 100 == 0 ? bytes("y".repeat(400)) : new byte[0]; + } - // Then — every input byte is represented. - assertThat(totalBytes(result)).isEqualTo(12); + // When + Sample result = Sample.draw(rows); + + // Then every line is a slice of a non-empty row, and the target is reached + assertThat(result.sampled()).isTrue(); + assertThat(totalBytes(result)).isGreaterThanOrEqualTo(Sample.TARGET_SAMPLE_BYTES); + for (int line = 0; line < result.lineCount(); line++) { + assertThat(result.lineEnd(line) - result.lineStart(line)).isPositive(); + } } @Test - void draw_allEmptyRows_producesEmptySample() { - // Given — every row empty, so there is nothing to sample. + void draw_emptyRowsOnly_givesAnEmptyLineEach() { + // Given every row empty byte[][] rows = {new byte[0], new byte[0], new byte[0]}; // When - Sample result = Sample.draw(rows, 1L); + Sample result = Sample.draw(rows); - // Then - assertThat(result.chunkCount()).isZero(); + // Then three empty lines, no bytes + assertThat(result.lineCount()).isEqualTo(3); assertThat(result.bytes()).isEmpty(); - assertThat(result.chunkCountForGeneration(0)).isZero(); } @Test - void draw_sameSeed_producesByteIdenticalSample() { - // Given - byte[][] rows = repeatedRows("deterministic sampling must reproduce ", 2_000); + void draw_longRows_takeLinesOfAtMostFiveHundredTwelveBytes() { + // Given rows of 5000 bytes each, 20 of them: 100KB, sampled + byte[][] rows = repeatedRows("x".repeat(5_000), 20); // When - Sample first = Sample.draw(rows, 99L); - Sample second = Sample.draw(rows, 99L); + Sample result = Sample.draw(rows); // Then - assertThat(second.bytes()).isEqualTo(first.bytes()); - assertThat(second.chunkCount()).isEqualTo(first.chunkCount()); + for (int line = 0; line < result.lineCount(); line++) { + assertThat(result.lineEnd(line) - result.lineStart(line)).isBetween(1, Sample.LINE_BYTES); + } } @Test - void chunkCountForGeneration_growsMonotonicallyToFullSample() { + void draw_sameRows_drawTheSameSampleAsTheContiguousOverload() { // Given - byte[][] rows = repeatedRows("grow the fraction over generations ", 2_000); - Sample sut = Sample.draw(rows, 7L); + byte[][] rows = repeatedRows("deterministic sampling must reproduce ", 2_000); // When - int gen0 = sut.chunkCountForGeneration(0); - int gen4 = sut.chunkCountForGeneration(4); + Sample perRow = Sample.draw(rows); + Sample contiguous = Sample.draw(flatten(rows), offsets(rows), rows.length); - // Then — the first generation sees only a fraction; the last sees every chunk. - assertThat(gen0).isGreaterThanOrEqualTo(1).isLessThan(sut.chunkCount()); - assertThat(gen4).isEqualTo(sut.chunkCount()); + // Then a fixed hash sequence, no seed: the same lines either way + assertThat(contiguous.bytes()).isEqualTo(perRow.bytes()); + assertThat(contiguous.lineCount()).isEqualTo(perRow.lineCount()); + } + + @Test + void hash_isRustsFsstHash() { + // Given Rust's `value.wrapping_mul(2971215073) ^ value.wrapping_shr(15)` + + // When + long zero = Sample.hash(0); + long one = Sample.hash(1); + long minusOne = Sample.hash(-1L); + + // Then + assertThat(zero).isZero(); + assertThat(one).isEqualTo(2971215073L); + assertThat(minusOne).isEqualTo(-2971215073L ^ (-1L >>> 15)); + } + + private static byte[] bytes(String text) { + return text.getBytes(StandardCharsets.UTF_8); } private static byte[][] repeatedRows(String text, int count) { - byte[] row = text.getBytes(StandardCharsets.UTF_8); + byte[] row = bytes(text); byte[][] rows = new byte[count][]; for (int i = 0; i < count; i++) { rows[i] = row; @@ -88,10 +128,32 @@ private static byte[][] repeatedRows(String text, int count) { return rows; } + private static byte[] flatten(byte[][] rows) { + int total = 0; + for (byte[] row : rows) { + total += row.length; + } + byte[] out = new byte[total]; + int pos = 0; + for (byte[] row : rows) { + System.arraycopy(row, 0, out, pos, row.length); + pos += row.length; + } + return out; + } + + private static int[] offsets(byte[][] rows) { + int[] out = new int[rows.length + 1]; + for (int i = 0; i < rows.length; i++) { + out[i + 1] = out[i] + rows[i].length; + } + return out; + } + private static int totalBytes(Sample sample) { int total = 0; - for (int i = 0; i < sample.chunkCount(); i++) { - total += sample.chunkEnd(i) - sample.chunkStart(i); + for (int line = 0; line < sample.lineCount(); line++) { + total += sample.lineEnd(line) - sample.lineStart(line); } return total; } diff --git a/fuzz/src/test/java/io/github/dfa1/vortex/fuzz/FsstFuzzTest.java b/fuzz/src/test/java/io/github/dfa1/vortex/fuzz/FsstFuzzTest.java index 3cd245389..5edbf256e 100644 --- a/fuzz/src/test/java/io/github/dfa1/vortex/fuzz/FsstFuzzTest.java +++ b/fuzz/src/test/java/io/github/dfa1/vortex/fuzz/FsstFuzzTest.java @@ -28,7 +28,7 @@ void compressDecompressRoundTrip(FuzzedDataProvider data) { for (int i = 0; i < rows.length; i++) { rows[i] = data.consumeBytes(data.consumeInt(0, 128)); } - Compressor compressor = new CompressorBuilder().seed(data.consumeLong()).train(rows); + Compressor compressor = new CompressorBuilder().train(rows); Decompressor decompressor = compressor.toDecompressor(); // A row the table was not trained on: all escapes is still a valid encoding. byte[] unseen = data.consumeRemainingAsBytes(); diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/EncodingTrees.java b/integration/src/test/java/io/github/dfa1/vortex/integration/EncodingTrees.java index da315b741..1a8c20943 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/EncodingTrees.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/EncodingTrees.java @@ -11,6 +11,7 @@ import java.io.IOException; import java.lang.foreign.MemorySegment; +import java.lang.foreign.ValueLayout; import java.nio.file.Path; import java.util.ArrayList; import java.util.List; @@ -36,6 +37,44 @@ static List withStats(Path file) throws IOException { return of(file, true); } + /// The raw buffers of the file's first flat segment, in the order its array node lists them, + /// without their alignment padding: what a trained FSST table and its code stream are, byte + /// for byte, whatever the cascade made of the children. + static List firstFlatBuffers(Path file) throws IOException { + try (VortexReader reader = VortexReader.open(file)) { + Layout flat = firstFlat(reader.layout()); + if (flat == null) { + throw new IllegalStateException("no flat layout in " + file); + } + SegmentSpec spec = reader.footer().segmentSpecs().get(flat.segments().getFirst()); + MemorySegment seg = reader.rawSegment(spec); + int fbLen = seg.get(LE_INT, seg.byteSize() - 4L); + FbsArray array = FbsArray.getRootAsFbsArray(seg.asSlice(seg.byteSize() - 4L - fbLen, fbLen)); + List out = new ArrayList<>(); + long offset = 0; + for (int i = 0; i < array.buffersLength(); i++) { + offset += array.buffers(i).padding(); + long length = array.buffers(i).length(); + out.add(seg.asSlice(offset, length).toArray(ValueLayout.JAVA_BYTE)); + offset += length; + } + return out; + } + } + + private static Layout firstFlat(Layout layout) { + if (layout.isFlat() && !layout.segments().isEmpty()) { + return layout; + } + for (Layout child : layout.children()) { + Layout found = firstFlat(child); + if (found != null) { + return found; + } + } + return null; + } + private static List of(Path file, boolean stats) throws IOException { List out = new ArrayList<>(); try (VortexReader reader = VortexReader.open(file)) { diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java new file mode 100644 index 000000000..ad27f79d4 --- /dev/null +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/FsstTrainingParityIntegrationTest.java @@ -0,0 +1,152 @@ +package io.github.dfa1.vortex.integration; + +import dev.vortex.api.Session; +import dev.vortex.api.VortexWriter; +import dev.vortex.arrow.ArrowAllocation; +import dev.vortex.jni.NativeLoader; +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.core.model.EncodingId; +import io.github.dfa1.vortex.writer.ColumnEncoding; +import io.github.dfa1.vortex.writer.WriteOptions; +import org.apache.arrow.c.ArrowArray; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.VarCharVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import java.io.IOException; +import java.nio.channels.FileChannel; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.function.IntFunction; +import java.util.stream.Stream; + +import static org.assertj.core.api.Assertions.assertThat; + +/// vortex-java trains the FSST symbol table as Rust's `fsst-rs` does (#583): the same sample, the same +/// five generations, the same candidates, gains and cut. The two tables are not byte for byte equal, +/// and cannot be: candidates of exactly equal gain leave Rust's heap in the order its hash map +/// iterates them, which depends on the CPU's SIMD width, so Rust's own table differs between x86 and +/// arm. What has to hold is that nearly every symbol is Rust's and the code stream is as long; +/// [NullableTextInteropIntegrationTest] holds a column whose whole encoding tree equals Rust's. +class FsstTrainingParityIntegrationTest { + + private static final Session SESSION = Session.create(); + private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator(); + private static final ColumnName V = ColumnName.of("v"); + + static { + NativeLoader.loadJni(); + } + + static Stream shapes() { + return Stream.of( + shape("urls", 5000, i -> "https://example.com/products/" + Long.toHexString(mix(i)) + "/detail?ref=" + i % 97), + shape("emails", 5000, i -> "user." + Long.toHexString(mix(i)) + "@mail-" + i % 17 + ".example.com"), + shape("urls small", 400, i -> "http://www.example.net/a/" + Long.toHexString(mix(i)) + "/index.html"), + // under 16 KiB: taken whole, and the last generation pruned + shape("emails whole", 250, i -> "user." + Long.toHexString(mix(i)) + "@mail-" + i % 17 + ".example.com")); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("shapes") + void javaTrainsAnFsstTableCloseToJnis(String name, int rows, IntFunction values, @TempDir Path tmp) + throws Exception { + // Given + Path jniFile = tmp.resolve("jni.vortex"); + Path javaFile = tmp.resolve("java.vortex"); + writeWithJni(jniFile, rows, values); + List tree = EncodingTrees.of(jniFile); + assertThat(tree).as("vortex-jni picked FSST for this column").anyMatch(l -> l.contains("vortex.fsst")); + + // When + writeWithJava(javaFile, rows, values); + + // Then + List expected = EncodingTrees.firstFlatBuffers(jniFile); + List actual = EncodingTrees.firstFlatBuffers(javaFile); + List rustSymbols = symbolSet(expected); + List javaSymbols = symbolSet(actual); + long common = javaSymbols.stream().filter(rustSymbols::contains).count(); + assertThat(common).as("symbols Rust also has, of %d (Java has %d)", rustSymbols.size(), javaSymbols.size()) + .isGreaterThanOrEqualTo((long) (0.8 * rustSymbols.size())); + assertThat((double) actual.get(2).length).as("code stream bytes, Rust's is %d", expected.get(2).length) + .isBetween(0.985 * expected.get(2).length, 1.015 * expected.get(2).length); + } + + /// The table's symbols as (length, bytes) strings, sorted: the table whatever order it is numbered in. + private static List symbolSet(List buffers) { + byte[] symbols = buffers.get(0); + byte[] lengths = buffers.get(1); + List out = new ArrayList<>(); + for (int i = 0; i < lengths.length; i++) { + StringBuilder hex = new StringBuilder(); + for (int k = 0; k < lengths[i]; k++) { + hex.append(String.format("%02x", symbols[i * 8 + k])); + } + out.add(hex.toString()); + } + Collections.sort(out); + return out; + } + + private static long mix(long i) { + long z = (i + 1) * 0x9E3779B97F4A7C15L; + z = (z ^ (z >>> 30)) * 0xBF58476D1CE4E5B9L; + z = (z ^ (z >>> 27)) * 0x94D049BB133111EBL; + return (z ^ (z >>> 31)) & Long.MAX_VALUE; + } + + private static Arguments shape(String name, int rows, IntFunction values) { + return Arguments.of(name, rows, values); + } + + private static void writeWithJni(Path path, int rows, IntFunction values) throws Exception { + Schema schema = new Schema(List.of(Field.notNullable("v", new ArrowType.Utf8()))); + try (VortexWriter writer = VortexWriter.builder(SESSION, path.toAbsolutePath().toUri().toString(), schema, + ALLOCATOR).build(); + VectorSchemaRoot root = VectorSchemaRoot.create(schema, ALLOCATOR)) { + VarCharVector vector = (VarCharVector) root.getVector("v"); + vector.allocateNew(); + for (int i = 0; i < rows; i++) { + vector.setSafe(i, values.apply(i).getBytes(StandardCharsets.UTF_8)); + } + vector.setValueCount(rows); + root.setRowCount(rows); + try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); + ArrowSchema arrowSchema = ArrowSchema.allocateNew(ALLOCATOR)) { + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, arrowSchema); + writer.writeBatch(array.memoryAddress(), arrowSchema.memoryAddress()); + } + } + } + + private static void writeWithJava(Path path, int rows, IntFunction values) throws IOException { + String[] strings = new String[rows]; + for (int i = 0; i < rows; i++) { + strings[i] = values.apply(i); + } + DType.Struct schema = new DType.Struct(List.of(V), List.of(DTypes.UTF8), false); + WriteOptions options = WriteOptions.defaults().withGlobalDict(false) + .withColumnEncoding(V, ColumnEncoding.candidates(EncodingId.VORTEX_FSST, EncodingId.VORTEX_VARBIN)); + try (var channel = FileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var writer = io.github.dfa1.vortex.writer.VortexWriter.create(channel, schema, options)) { + writer.writeChunk(Map.of(V, strings)); + } + } +} diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java index 01166aac3..978521d6c 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java @@ -86,7 +86,7 @@ static Stream shapes() { /// The shapes whose whole tree is Rust's. The others still differ beyond the validity, in the /// scheme chosen for the text itself (#650: the probe applies to numbers only; FSST's lengths). static Stream treeShapes() { - return shapes().filter(a -> a.get()[0].equals("unique, no nulls")); + return shapes().filter(a -> a.get()[0].equals("unique, no nulls") || a.get()[0].equals("few distinct, no nulls")); } @ParameterizedTest(name = "{0}") diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/FsstEncodingEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/FsstEncodingEncoder.java index c1da9ef9d..a6c953746 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/FsstEncodingEncoder.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/FsstEncodingEncoder.java @@ -32,19 +32,16 @@ /// /// Symbols are laid out on the wire in length order — all multi-byte symbols (length 2-8) first in /// non-decreasing length order, then all length-1 symbols — as the `vortex.fsst` wire contract -/// requires (mirroring Rust `FSSTData::validate_symbol_lengths`). The [Compressor] instead numbers -/// its codes in gain-descending order, so this adapter remaps every code the compressor emits into -/// that length-sorted wire order (see [#encode]). +/// requires (mirroring Rust `FSSTData::validate_symbol_lengths`), and within the 2-byte ones as Rust's +/// `fsst-rs` numbers them ([Compressor#codesSortedByLength()]). The [Compressor] numbers its codes in +/// the order it took them, so this adapter remaps every code the compressor emits into that wire +/// order (see [#encode]). public final class FsstEncodingEncoder implements EncodingEncoder { /// This encoding, barred from its own children: the lengths and code offsets are primitive /// arrays FSST cannot compress anyway, and the slots exist for bitpacking/constant folding. private static final Set SELF = Set.of(EncodingId.VORTEX_FSST); - /// Fixed seed for the training-sample PRNG. Encoding must be reproducible: the same input - /// always trains the same symbol table and produces byte-identical output. - private static final long TRAINING_SAMPLE_SEED = 0x5EEDL; - /// Creates the encoder. public FsstEncodingEncoder() { } @@ -173,8 +170,7 @@ private static Fsst compress(Object data, VarBinBytes.Rows rows, Arena arena) { maxUncompLen = Math.max(maxUncompLen, uncompLens[i]); } - Compressor trained = new CompressorBuilder().seed(TRAINING_SAMPLE_SEED) - .train(rowBytes, rowOffsets, n); + Compressor trained = new CompressorBuilder().train(rowBytes, rowOffsets, n); // byte[]-input overload over the shared row buffer: the intrinsified VarHandle word load // is faster than wrapping each row in a MemorySegment first, and the row is addressed as @@ -201,7 +197,7 @@ static Fsst compressPerRow(byte[][] byteArrays, Arena arena) { maxUncompLen = Math.max(maxUncompLen, uncompLens[i]); } - Compressor trained = new CompressorBuilder().seed(TRAINING_SAMPLE_SEED).train(byteArrays); + Compressor trained = new CompressorBuilder().train(byteArrays); return finishCompress(trained, n, totalInput, maxUncompLen, uncompLens, arena, (compressor, i, scratch, destOffset) -> { @@ -229,8 +225,8 @@ private static Fsst finishCompress(Compressor trained, int n, long totalInput, i int numSymbols = trained.symbolCount(); // Wire order: the wire lists symbols in length order (multi-byte length-ascending, then - // length-1 last), whereas training numbers its codes gain-descending. wireOrder[i] is the - // trained (internal) code that belongs at wire position i. + // length-1 last), whereas training numbers its codes in the order it took them. + // wireOrder[i] is the trained (internal) code that belongs at wire position i. int[] wireOrder = trained.codesSortedByLength(); MemorySegment symBuf = arena.allocate(numSymbols * 8L, 8);