Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- `EncodeContext` has a new last component, `trace`, so code calling its canonical constructor passes `EncodeTrace.NONE`. ([#601](https://github.com/dfa1/vortex-java/issues/601))

### Removed
- `CompressorBuilder#seed(long)` of the `vortex-fsst` module: training takes Rust's fixed sample, not a seeded one. ([#583](https://github.com/dfa1/vortex-java/issues/583))

### Fixed
- FSST symbol tables are trained as Rust's `fsst-rs` does (its sample, generations, candidate gains and cut), so a text column without a null gets Rust's FSST encoding up to ties of equal gain, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650))
- A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696))

Expand Down
2 changes: 1 addition & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) |
| `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct |
| `vortex.chunked` | `ChunkedEncodingDecoder` | `ChunkedEncodingEncoder` | ✅ | ✅ | Primitive + Struct concat |
| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary |
| `vortex.fsst` | `FsstEncodingDecoder` | `FsstEncodingEncoder` | ✅ | ✅ | Utf8, Binary. The symbol table is trained as Rust's `fsst-rs` v0.6.0 does ([#583](https://github.com/dfa1/vortex-java/issues/583)): its sample, five generations, candidate gains, cut and final code order. Not byte for byte Rust's, by decision: candidates of equal gain and length pop in the order they were counted, where Rust pops them in its hash map's iteration order, which depends on the CPU's SIMD width (Rust's own table differs between x86 and arm); about 90% of the symbols and a code stream within 1% are Rust's. A column with nulls trains on its empty placeholder rows, where Rust trains on the valid rows only, until its validity lives in the encoding (#650) |
| `vortex.list` | `ListEncodingDecoder` | `ListEncodingEncoder` | ✅ | ✅ | A nullable list's validity is its optional third child, as in Rust |
| `vortex.listview` | `ListViewEncodingDecoder` | `ListViewEncodingEncoder` | ✅ | ✅ | Validity lives in the encoding's own fourth child slot, not under a `vortex.masked` wrapper. Read: both shapes decode. Write: only `vortex.map`'s entries child takes this path — a plain nullable `DType.List` column still routes through `vortex.masked` + `vortex.list` |
| `vortex.map` | `MapEncodingDecoder` | `MapEncodingEncoder` | ✅ | ✅ | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately |
Expand Down
8 changes: 4 additions & 4 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,9 +423,9 @@ zero-ceremony standalone use, and both paths produce identical output for the sa

| Method | Notes |
|-------------------------|-----------------------------------------------------------------------------|
| `new CompressorBuilder()` | Fresh builder using a fixed reproducible default seed |
| `seed(long)` | Override the training-sample seed; returns `this` for chaining |
| `train(byte[][] rows)` | Runs the bottom-up generation loop, returns a trained `Compressor` (deterministic in rows + seed) |
| `new CompressorBuilder()` | A builder; training takes no options: it follows Rust's `fsst-rs`, its sample included |
| `train(byte[][] rows)` | Runs the bottom-up generation loop, returns a trained `Compressor` (deterministic in the rows) |
| `train(byte[] rowBytes, int[] rowOffsets, int rowCount)` | The same over rows held contiguously, as a varbin column has them |

### `Compressor`

Expand All @@ -437,7 +437,7 @@ Immutable trained symbol table (up to 255 codes; `0xFF` is the escape). Produced
| `symbolCount()` | Number of symbols in the table (0–255) |
| `packedSymbol(int code)` | The symbol's bytes packed LSB-first into a `long` |
| `symbolLength(int code)` | The symbol's length in bytes (1–8) |
| `codesSortedByLength()` | Code permutation, multi-byte length-ascending then all length-1 last |
| `codesSortedByLength()` | Code permutation in Rust's final order: 2-byte symbols no longer symbol starts with, those one does (last taken first), 3–8 bytes by length, then the 1-byte ones |
| `toDecompressor()` | A `Decompressor` bound to this table |
| `compress(byte[] in, int start, int end, byte[] out, long outPos)` | Greedy longest-match compress; size `out` ≥ `2 * (end - start)` |
| `compress(MemorySegment in, long start, long end, MemorySegment out, long outPos)` | FFM-native hot path; same sizing rule |
Expand Down
92 changes: 64 additions & 28 deletions fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,27 @@ public final class Compressor {
static final VarHandle LONG_LE_BYTES =
MethodHandles.byteArrayViewVarHandle(long[].class, ByteOrder.LITTLE_ENDIAN);

/// Slots of the symbol table Rust's builder keeps, whose real symbols are the first `symbolCount()`.
private static final int SYMBOL_SLOTS = 255;

private final List<Symbol> symbolsByGainDescending;
private final Matcher matcher;
/// The bytes of each of the table's [#SYMBOL_SLOTS] slots as Rust's builder holds them: the real
/// symbols, then, past them, whatever an earlier generation left there, which Rust never clears.
private final long[] slots;

private Compressor(List<Symbol> symbolsByGainDescending, Matcher matcher) {
private Compressor(List<Symbol> symbolsByGainDescending, Matcher matcher, long[] slots) {
this.symbolsByGainDescending = symbolsByGainDescending;
this.matcher = matcher;
this.slots = slots;
}

private static long[] slotsOf(List<Symbol> symbols, long[] previous) {
long[] slots = previous.clone();
for (int code = 0; code < symbols.size(); code++) {
slots[code] = symbols.get(code).packedBytes();
}
return slots;
}

/// Creates a compressor over the given symbols, which must be in gain-descending order (code =
Expand All @@ -51,7 +66,7 @@ private Compressor(List<Symbol> symbolsByGainDescending, Matcher matcher) {
/// @return a compressor bound to that table
static Compressor of(List<Symbol> symbolsByGainDescending) {
List<Symbol> copy = List.copyOf(symbolsByGainDescending);
return new Compressor(copy, Matcher.of(copy));
return new Compressor(copy, Matcher.of(copy), slotsOf(copy, new long[SYMBOL_SLOTS]));
}

/// Rebuilds a compressor from new symbols, reusing `previous`'s [Matcher] backing arrays instead
Expand All @@ -64,7 +79,7 @@ static Compressor of(List<Symbol> symbolsByGainDescending) {
/// matcher arrays
static Compressor rebuild(List<Symbol> symbolsByGainDescending, Compressor previous) {
List<Symbol> copy = List.copyOf(symbolsByGainDescending);
return new Compressor(copy, Matcher.rebuild(copy, previous.matcher));
return new Compressor(copy, Matcher.rebuild(copy, previous.matcher), slotsOf(copy, previous.slots));
}

/// Returns an equivalent compressor whose codes are permuted: the symbol currently at code
Expand Down Expand Up @@ -119,49 +134,70 @@ public int symbolLength(int code) {
return symbolsByGainDescending.get(code).length();
}

/// Returns a permutation of the codes ordered by symbol length: all multi-byte symbols (length
/// 2-8) first in non-decreasing length order, then all length-1 symbols last.
/// Returns a permutation of the codes in the order Rust's `fsst-rs` numbers them once its table is
/// final (`finalize`): the 2-byte symbols that no other symbol shares its first two bytes with, in
/// the order they were taken; then the 2-byte symbols some longer symbol starts with, last taken
/// first; then the 3, 4, ..., 8-byte symbols, each length in the order they were taken; then the
/// 1-byte symbols.
///
/// This is pure, wire-format-agnostic reordering logic — "sort by length, single-byte-last" —
/// with no `vortex.fsst`-specific naming or semantics baked in. The `vortex.fsst` wire adapter
/// (a separate module) happens to interpret this exact permutation as its on-wire symbol order,
/// but that interpretation lives entirely in the adapter; the algorithm module only knows how to
/// sort. Ties within a length bucket keep their gain-descending relative order (a stable sort),
/// so the permutation is deterministic.
/// The split of the 2-byte symbols lets Rust's compressor skip its hash table for a 2-byte code
/// below the split, which no longer symbol can extend. Rust asks "does another symbol start with
/// these two bytes" of every slot of its table, including the ones an earlier training generation
/// left behind past the final symbols, so a table smaller than an earlier one splits differently
/// than its own symbols alone would.
///
/// @return an `int[]` permutation of `0 .. symbolCount() - 1` in length-sorted, single-byte-last
/// order
/// @return an `int[]` permutation of `0 .. symbolCount() - 1`: the code at each position
public int[] codesSortedByLength() {
// Counting sort over symbol length: lengths are bounded to 1-8 (Matcher.maxSymbolLength()),
// so a comparator-based sort is unnecessary work — this is O(symbolCount) with no boxing,
// and a single forward pass over codes 0..symbolCount()-1 (their gain-descending order) into
// each length's bucket is stable by construction, with no comparator needed to break ties.
int maxSymbolLength = Matcher.maxSymbolLength();
int[] countByLength = new int[maxSymbolLength + 1]; // index 1..maxSymbolLength; 0 unused
int n = symbolCount();
int[] countByLength = new int[maxSymbolLength + 1]; // index 1..maxSymbolLength; 0 unused
for (int code = 0; code < n; code++) {
countByLength[symbolLength(code)]++;
}

// Bucket start offsets in wire order: multi-byte lengths 2..maxSymbolLength ascending, then
// length-1 last.
int[] bucketStart = new int[maxSymbolLength + 1];
int offset = 0;
for (int length = 2; length <= maxSymbolLength; length++) {
bucketStart[length] = offset;
offset += countByLength[length];
// First position of each length: 1-byte symbols after all the others; the 2-byte ones from 0,
// split at the suffix limit; then each longer length after the one before.
int[] nextPosition = new int[maxSymbolLength + 1];
nextPosition[1] = n - countByLength[1];
nextPosition[2] = 0;
for (int length = 2; length < maxSymbolLength; length++) {
nextPosition[length + 1] = nextPosition[length] + countByLength[length];
}
bucketStart[1] = offset;
int noSuffix = 0;
int hasSuffix = nextPosition[3];

int[] result = new int[n];
int[] nextSlot = bucketStart.clone();
for (int code = 0; code < n; code++) {
int length = symbolLength(code);
result[nextSlot[length]++] = code;
int position;
if (length == 2) {
if (sharesFirstTwoBytes(code)) {
hasSuffix--;
position = hasSuffix;
} else {
position = noSuffix++;
}
} else {
position = nextPosition[length]++;
}
result[position] = code;
}
return result;
}

/// Whether another slot starts with the same two bytes as the 2-byte symbol `code`: as Rust's
/// `finalize` asks it, of every slot of its table, the ones past the real symbols too, with what
/// an earlier generation left in them.
private boolean sharesFirstTwoBytes(int code) {
long first2 = slots[code] & 0xFFFF;
for (int other = 0; other < SYMBOL_SLOTS; other++) {
if (other != code && (slots[other] & 0xFFFF) == first2) {
return true;
}
}
return false;
}

/// Builds a [Decompressor] over this compressor's table so the same trained symbols round-trip
/// without re-deriving the parallel arrays at the call site.
///
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,22 @@

import java.util.List;

/// Trains an FSST [Compressor] from a corpus of rows using the FSST paper's bottom-up algorithm
/// (Algorithm 3) plus the Rust reference's four engineering refinements (issue #287).
/// Trains an FSST [Compressor] from a corpus of rows as Rust's `fsst-rs` does (`Compressor::train`,
/// the FSST paper's Algorithm 3 with the C++ reference's refinements): the same sample, the same five
/// generations over a growing share of its lines, and the same candidates, gains and cut.
///
/// Training runs [#GENERATIONS] generations starting from an empty symbol table. Each generation
/// compress-counts a growing fraction of one fixed [Sample] with the current table, then rebuilds
/// the table from the top-gain candidates (see [TrainingGeneration]). The refinements — adaptive
/// per-generation sample fraction, per-generation min-count pruning, the 8x single-byte gain boost,
/// and a final cost-based prune — are all applied inside that per-generation step; this class only
/// drives the loop and threads the seed through.
///
/// Training is deterministic: the same rows and seed always produce a byte-identical [Compressor],
/// so an encoder built on top of it produces byte-identical output for a given input. Use the
/// builder to override the seed:
/// Training is deterministic: the same rows always produce a byte-identical [Compressor]. Rust's own
/// tables differ from this one only in where candidates of exactly equal gain and length are
/// ordered, which depends on its hash map ([TrainingGeneration]).
///
/// ```java
/// Compressor compressor = new CompressorBuilder().seed(42L).train(rows);
/// Compressor compressor = new CompressorBuilder().train(rows);
/// ```
public final class CompressorBuilder {

/// Number of training generations. The FSST paper reports the table stabilizing after roughly
/// five passes; each pass can pair adjacent symbols to roughly double their length, so five
/// passes grow symbols to the full 8-byte length from an empty start.
private static final int GENERATIONS = Sample.SAMPLE_FRACTION_NUMERATORS.length;

/// Default PRNG seed when [#seed(long)] is not called. A fixed constant (not wall-clock derived)
/// keeps training reproducible out of the box.
private static final long DEFAULT_SEED = 0x5EEDL;

private long seed = DEFAULT_SEED;

/// Creates a builder using the default seed until [#seed(long)] is called.
/// Creates a builder.
public CompressorBuilder() {
// Explicit no-arg constructor: fields default to the reproducible DEFAULT_SEED.
}

/// Sets the PRNG seed used to draw the training sample. The same rows and seed always train a
/// byte-identical compressor.
///
/// @param seed the sample-drawing seed
/// @return this builder, for chaining
public CompressorBuilder seed(long seed) {
this.seed = seed;
return this;
// Explicit no-arg constructor: nothing to configure, Rust's training takes no options.
}

/// Trains a compressor over `rows`, running the full bottom-up generation loop.
Expand All @@ -56,38 +29,37 @@ public CompressorBuilder seed(long seed) {
/// empty)
/// @return the trained compressor
public Compressor train(byte[][] rows) {
return trainOn(Sample.draw(rows, seed));
if (rows.length == 0) {
return Compressor.of(List.of());
}
return trainOn(Sample.draw(rows));
}

/// Trains a compressor over rows held contiguously, where row `i` spans
/// `rowBytes[rowOffsets[i], rowOffsets[i + 1])`.
///
/// Equivalent to [#train(byte[][])] — same sample, same resulting table for the same seed —
/// but takes the corpus in the contiguous shape a varbin column already has, so the caller
/// need not materialize one `byte[]` per row just to train.
/// Equivalent to [#train(byte[][])] (same sample, same table) but takes the corpus in the
/// contiguous shape a varbin column already has, so the caller need not materialize one `byte[]`
/// per row just to train.
///
/// @param rowBytes all rows' bytes concatenated
/// @param rowOffsets `rowCount + 1` cumulative offsets into `rowBytes`
/// @param rowCount the number of rows
/// @return the trained compressor
public Compressor train(byte[] rowBytes, int[] rowOffsets, int rowCount) {
return trainOn(Sample.draw(rowBytes, rowOffsets, rowCount, seed));
if (rowCount == 0) {
return Compressor.of(List.of());
}
return trainOn(Sample.draw(rowBytes, rowOffsets, rowCount));
}

private Compressor trainOn(Sample sample) {
Compressor compressor = Compressor.of(List.of());
if (sample.chunkCount() == 0) {
return compressor;
}
// Reused across every generation instead of allocating fresh count arrays five times over
// (issue #395).
TrainingGeneration.Counts counts = new TrainingGeneration.Counts();
for (int gen = 0; gen < GENERATIONS; gen++) {
boolean finalGeneration = gen == GENERATIONS - 1;
int chunkLimit = sample.chunkCountForGeneration(gen);
int fractionNumerator = Sample.SAMPLE_FRACTION_NUMERATORS[gen];
List<Symbol> symbols = TrainingGeneration.run(
compressor, sample, chunkLimit, fractionNumerator, finalGeneration, counts);
for (int fraction : Sample.FRACTION_NUMERATORS) {
List<Symbol> symbols = TrainingGeneration.run(compressor, sample, fraction, counts);
// Reuse compressor's own (now fully consumed) matcher arrays instead of allocating a
// fresh ~288 KB pair of tables on every one of the five generations (issue #393 #7).
compressor = Compressor.rebuild(symbols, compressor);
Expand Down
Loading
Loading