diff --git a/CHANGELOG.md b/CHANGELOG.md index 194147ee2..112dc987a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The child nodes of a flat segment carry the array stats Rust keeps: a sequence its sorted flags, a nullable integer's validity bitmap its true count, and patch values `is_constant = false`. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - The zone table of a column holds Rust's shape: no validity or mask on a column without nulls, and a constant bool stays a `vortex.bool` until the constant is smaller. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - Text columns stay `vortex.varbinview`, Rust's canonical form, unless a scheme is smaller, and a `vortex.varbin` is only chosen for binary. ([#583](https://github.com/dfa1/vortex-java/issues/583)) +- A numeric column keeps its dict layout past 2,048 distinct values, admitted by Rust's probe on its first 8,192 rows and starting a new dictionary at 65,535 entries as Rust does. ([#649](https://github.com/dfa1/vortex-java/issues/649)) - A text column keeps its dict layout past 32,768 distinct values, starting a new dictionary at 65,535 entries or 1 MiB as Rust does, and is admitted by its first 8,192 rows. ([#650](https://github.com/dfa1/vortex-java/issues/650)) - The compact preset writes text columns as zstd frames of 8192 values, as Rust does, so a row-range scan decompresses only the frames it overlaps. ([#636](https://github.com/dfa1/vortex-java/issues/636)) - A nullable decimal column keeps its validity in its own encoding, or has none when every row is valid, instead of a `vortex.masked` wrapper, and a column of equal decimals is one constant, as Rust writes them: decimal zone-stats tables are smaller. ([#646](https://github.com/dfa1/vortex-java/issues/646)) diff --git a/docs/compatibility.md b/docs/compatibility.md index f8b6feee0..16647f6ad 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -125,7 +125,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul | `vortex.varbinview` | `VarBinViewEncodingDecoder` | `VarBinViewEncodingEncoder` | ✅ | ✅ | Utf8, Binary | | `vortex.alp` | `AlpEncodingDecoder` | `AlpEncodingEncoder` | ✅ | ✅ | F64, F32 | | `vortex.alprd` | `AlpRdEncodingDecoder` | `AlpRdEncodingEncoder` | ✅ | ✅ | F64, F32 | -| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and under half distinct values in its first batch, at most 2,048 of them for numbers (Rust has no such cap and skips only above half); a text column has no cap either: it needs under half of its first 8,192 rows distinct, and starts a new dictionary whenever the open one reaches 65,535 entries or 1 MiB, as Rust's dict layout does. A non-null primitive column that passes then gets the layout only when the cascade, with the integer dictionary back in, picks `vortex.dict` for its first rows: Rust's `DictStrategy` probe. Rust probes exactly the first 8192 rows across batches; Java probes the first batch's first 8192, so a first batch under 8192 rows can decide differently. Nullable and string columns skip the probe and decide by the distinct-count check alone, which admits some nullable integer columns Rust bit-packs. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust | +| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and Rust's probe: the cascade, run on the column's first 8,192 rows, must pick `vortex.dict` (Rust's `DictStrategy`). Text columns skip the probe, as our cascade picks OnPair where Rust's string dictionary wins, and need under half of those rows distinct. Neither has a cardinality cap: whenever the open dictionary reaches 65,535 entries or 1 MiB the writer closes it and starts the next, and a column of several dictionaries is a chunked layout of dict layouts, as in Rust. With `cascading(0)` there is no cascade to probe, so the distinct count of those rows (under half, and more than one) decides. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust | | `vortex.sparse` | `SparseEncodingDecoder` | `SparseEncodingEncoder` | ✅ | ✅ | Primitive, Bool, Utf8/Binary (decode); Primitive (encode) | | `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) | | `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct | diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutStringInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java similarity index 59% rename from integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutStringInteropIntegrationTest.java rename to integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java index b5f17f885..c7257126e 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutStringInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java @@ -21,6 +21,7 @@ import org.apache.arrow.vector.types.pojo.ArrowType; import org.apache.arrow.vector.types.pojo.Field; import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import org.junit.jupiter.params.ParameterizedTest; @@ -38,11 +39,12 @@ import static org.assertj.core.api.Assertions.assertThat; -/// Which text columns get Rust's dict layout, and how many dictionaries it makes (#650). Rust's -/// `DictStrategy` admits a column by its first 8192 rows (not more than half distinct) and starts a -/// new dictionary whenever the open one reaches 65,535 entries or 1 MiB, so a column of any total -/// cardinality can keep its dict layout. -class DictLayoutStringInteropIntegrationTest { +/// Which text and numeric columns get Rust's dict layout, and how many dictionaries it makes (#650, +/// #649). Rust's `DictStrategy` admits a column by its first 8192 rows and starts a new dictionary +/// whenever the open one reaches 65,535 entries or 1 MiB, so a column of any total cardinality can +/// keep its dict layout. +@Tag("slow") +class DictLayoutInteropIntegrationTest { private static final Session SESSION = Session.create(); private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator(); @@ -100,6 +102,96 @@ void startsANewDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws Exceptio assertThat(readAll(java)).containsExactly(values); } + @ParameterizedTest(name = "pool of {0} values") + @ValueSource(ints = {100, 3_000, 20_000, 150_000}) + void admitsTheNumericColumnsRustAdmits(int poolSize, @TempDir Path tmp) throws Exception { + // Given — 3 chunks of full-width i64 drawn at random from a pool (hashes, like ClickBench's URLHash) + Random r = new Random(11); + long[] pool = new long[poolSize]; + for (int i = 0; i < pool.length; i++) { + pool[i] = r.nextLong(); + } + long[] values = new long[CHUNK * 3]; + for (int i = 0; i < values.length; i++) { + values[i] = pool[r.nextInt(pool.length)]; + } + + // When + Path java = writeJava(tmp, values); + Path jni = writeJni(tmp, values); + + // Then + assertThat(dictLayouts(EncodingTrees.of(java))).isEqualTo(dictLayouts(EncodingTrees.of(jni))); + } + + @Test + void startsANewNumericDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws Exception { + // Given — 8 chunks: the first 8192 rows hold 2000 distinct values, every row after them is new + Random r = new Random(5); + long[] values = new long[CHUNK * 8]; + for (int i = 0; i < values.length; i++) { + values[i] = i < 8_192 ? r.nextLong() % 2_000 * 1_000_003L : r.nextLong(); + } + + // When + Path java = writeJava(tmp, values); + Path jni = writeJni(tmp, values); + + // Then — whatever Rust decides for this column, so do we, dictionary for dictionary + assertThat(dictLayouts(EncodingTrees.of(java))).isEqualTo(dictLayouts(EncodingTrees.of(jni))); + assertThat(readAllLongs(java)).containsExactly(values); + } + + private static Path writeJava(Path tmp, long[] values) throws IOException { + Path file = tmp.resolve("java.vortex"); + DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.I64), false); + try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var w = io.github.dfa1.vortex.writer.VortexWriter.create(ch, schema, WriteOptions.defaults())) { + for (int from = 0; from < values.length; from += CHUNK) { + w.writeChunk(Map.of(ColumnName.of("url"), java.util.Arrays.copyOfRange(values, from, from + CHUNK))); + } + } + return file; + } + + private static Path writeJni(Path tmp, long[] values) throws Exception { + Path file = tmp.resolve("jni.vortex"); + Schema arrow = new Schema(List.of(Field.notNullable("url", new ArrowType.Int(64, true)))); + try (VortexWriter w = VortexWriter.builder(SESSION, file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) { + for (int from = 0; from < values.length; from += CHUNK) { + try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) { + org.apache.arrow.vector.BigIntVector v = (org.apache.arrow.vector.BigIntVector) root.getVector("url"); + root.allocateNew(); + for (int i = 0; i < CHUNK; i++) { + v.setSafe(i, values[from + i]); + } + root.setRowCount(CHUNK); + try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema as = ArrowSchema.allocateNew(ALLOCATOR)) { + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, as); + w.writeBatch(array.memoryAddress(), as.memoryAddress()); + } + } + } + } + return file; + } + + private static long[] readAllLongs(Path file) throws IOException { + List out = new ArrayList<>(); + try (var reader = VortexReader.open(file, ReadRegistry.loadAll()); + var iter = reader.scan(ScanOptions.columns(ColumnName.of("url")))) { + while (iter.hasNext()) { + try (var chunk = iter.next()) { + var longs = (io.github.dfa1.vortex.reader.array.LongArray) chunk.column(ColumnName.of("url")); + for (long i = 0; i < longs.length(); i++) { + out.add(longs.getLong(i)); + } + } + } + } + return out.stream().mapToLong(Long::longValue).toArray(); + } + private static Path writeJava(Path tmp, String[] values) throws IOException { Path file = tmp.resolve("java.vortex"); DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.UTF8), false); diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java b/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java index ffd29c2ca..bab51b91b 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java @@ -21,7 +21,7 @@ /// Cardinality-bounded buffering state for one global-dictionary candidate column (ADR 0021). Instead of /// retaining raw values from a column's first chunk until `close()`, this holds a deduplicated -/// value-to-code map (first-seen order, capped at [#GLOBAL_DICT_MAX_CARDINALITY]) and one cheap +/// value-to-code map (first-seen order, at most [#MAX_ENTRIES] entries per dictionary) and one cheap /// `short[]` code array per ingested chunk. Per-chunk stats are captured at ingest time from the /// raw chunk, before it is discarded. /// @@ -31,14 +31,6 @@ /// therefore the wire codes. final class DictColumnState { - // Columns with global cardinality below this threshold are dict-encoded across all chunks. - // The cap is type-aware. Numeric stays low: a global dict hurts high-cardinality F64/I64 - // columns (ALP/bitpacked codes beat U16 dict codes). Utf8/Binary is raised far higher — text columns - // with thousands of repeated distinct values (street/place names) dictionary-compress well - // (#299), and the per-chunk short[] code buffer holds codes 0..32767 (up to 32768 distinct) - // with no wider buffer; the codes are always U16 (CODES_PTYPE). - static final int GLOBAL_DICT_MAX_CARDINALITY = 2_048; - /// The codes ptype of every global dict: Rust's `DictLayoutConstraints` derive it from the /// configured maximum dictionary length, not the actual one -- U8 only when `max_len <= 255`, /// and the default `max_len` is `u16::MAX` -- so a default Rust global dict always has U16 @@ -47,11 +39,11 @@ final class DictColumnState { /// U16 codes it does not (taxi Airport_fee). static final PType CODES_PTYPE = PType.U16; - /// Rust's `DictLayoutConstraints` defaults for a Utf8/Binary dictionary: at most `u16::MAX` - /// entries and 1 MiB of views plus out-of-line bytes. A column that reaches either starts a new - /// dictionary, so a text column's cardinality is not a reason to give up the dict layout. - static final int MAX_VARBIN_ENTRIES = 65_535; - static final long MAX_VARBIN_DICT_BYTES = 1L << 20; + /// Rust's `DictLayoutConstraints` defaults: at most `u16::MAX` entries and 1 MiB of values (of + /// views plus out-of-line bytes for text). A column that reaches either starts a new dictionary, + /// so a column's cardinality is not a reason to give up the dict layout. + static final int MAX_ENTRIES = 65_535; + static final long MAX_DICT_BYTES = 1L << 20; private static final int VIEW_BYTES = 16; private static final int MAX_INLINED_BYTES = 12; @@ -209,53 +201,44 @@ Object uniques() { }; } - /// Ingests one primitive chunk into this candidate column's cardinality-bounded dict state (ADR - /// 0021): dedups each valid value into the shared value-to-code map, appends a per-chunk `short[]` - /// code array, and captures the chunk's row/null counts and min/max/sum stats before the raw array - /// is discarded. Null slots buffer the null entry's code, assigned at the first one. - /// - /// Returns `false` — without mutating this state — the moment a new distinct value would push the - /// map past [#GLOBAL_DICT_MAX_CARDINALITY]; the caller then demotes the column to per-chunk - /// encoding. This moves the cap check from `close()` to a continuous, mid-file guard so a column - /// whose distinct set grows past the cap never accumulates unbounded memory first. + /// Ingests the leading rows of a primitive chunk that fit this dictionary under Rust's constraints + /// (`DictLayoutConstraints`: [#MAX_ENTRIES] entries and 1 MiB of values), as Rust's primitive dict + /// builder does: the first entry is always admitted, and the row that needs an entry that no + /// longer fits ends the prefix. The caller starts a new dictionary with the rest. Dedups each + /// valid value into the shared value-to-code map, appends a per-chunk `short[]` code array, and + /// captures the prefix's row/null counts and min/max/sum stats before the raw array is discarded. + /// Null slots buffer the null entry's code, assigned at the first one. /// /// @param data the chunk data (a primitive array or a [NullableData] wrapper) - /// @return `true` if the chunk was ingested within the cardinality cap; `false` if the column - /// must be demoted - boolean ingestDictChunk(Object data) { + /// @return how many leading rows were ingested, `0` when the first row already needs a new + /// dictionary + int ingestDictChunk(Object data) { boolean nullableData = data instanceof NullableData; Object values = nullableData ? ((NullableData) data).values() : data; boolean[] validity = nullableData ? ((NullableData) data).validity() : null; int len = primitiveArrayLen(values, ptype); - int cap = GLOBAL_DICT_MAX_CARDINALITY; - int startSize = valueToCode.size(); - int startNullCode = nullCode; - - // One pass: insert new values and build the per-chunk code array. Ingest stays - // all-or-nothing — a chunk that would breach the cap rolls back the entries it added (the - // map's tail, codes >= startSize) — so the demoting caller still sees untouched state. The - // earlier shape ran a non-mutating counting pass first to get that atomicity, which cost a - // second boxed map probe for every row in the file. + int cap = (int) Math.min(MAX_ENTRIES, MAX_DICT_BYTES / ptype.byteSize()); + short[] codes = new short[len]; // Probe the unboxed bits index and box only when inserting a value the dictionary has not - // seen before (at most `cap` times for the whole file). The boxed key itself still comes - // from the source array, so I32 vs I64 and NaN/-0.0 keying stay exactly what a - // Double/Long-keyed map gave. + // seen before. The boxed key itself still comes from the source array, so I32 vs I64 and + // NaN/-0.0 keying stay exactly what a Double/Long-keyed map gave. long[] bits = rawBits(values, ptype, len); + int consumed = len; for (int i = 0; i < len; i++) { if (validity != null && !validity[i]) { if (nullCode < 0 && !addNullEntry(cap)) { - rollbackTo(startSize, startNullCode); - return false; + consumed = i; + break; } codes[i] = (short) nullCode; continue; } int code = bitsIndex.get(bits[i]); if (code < 0) { - if (valueToCode.size() == cap) { - rollbackTo(startSize, startNullCode); - return false; + if (!valueToCode.isEmpty() && valueToCode.size() >= cap) { + consumed = i; + break; } code = valueToCode.size(); valueToCode.put(readPrimitiveElement(values, ptype, i), code); @@ -263,15 +246,31 @@ boolean ingestDictChunk(Object data) { } codes[i] = (short) code; } - + if (consumed == 0) { + return 0; + } + if (consumed < len) { + codes = Arrays.copyOf(codes, consumed); + validity = validity != null ? Arrays.copyOf(validity, consumed) : null; + values = prefix(values, consumed); + } MinMax mm = PrimitiveEncodingEncoder.minMaxStats(ptype, values); - record(codes, validity, len, mm != null ? mm.min() : null, mm != null ? mm.max() : null, + record(codes, validity, consumed, mm != null ? mm.min() : null, mm != null ? mm.max() : null, PrimitiveEncodingEncoder.sumStat(ptype, values)); - return true; + return consumed; + } + + private static Object prefix(Object values, int n) { + return switch (values) { + case int[] a -> Arrays.copyOf(a, n); + case long[] a -> Arrays.copyOf(a, n); + case double[] a -> Arrays.copyOf(a, n); + default -> throw new IllegalStateException("not a global dict carrier: " + values.getClass()); + }; } /// Ingests the leading rows of a text chunk that fit this dictionary under Rust's constraints - /// (`DictLayoutConstraints`: [#MAX_VARBIN_ENTRIES] entries, [#MAX_VARBIN_DICT_BYTES] bytes), as + /// (`DictLayoutConstraints`: [#MAX_ENTRIES] entries, [#MAX_DICT_BYTES] bytes), as /// Rust's bytes dict builder does: the first entry is always admitted, and the row that needs an /// entry that no longer fits ends the prefix. The caller starts a new dictionary with the rest. /// @@ -308,7 +307,7 @@ int ingestVarBinChunk(Object data) { if (code == null) { long added = entryBytes(v); boolean first = valueToCode.isEmpty(); - if (!first && (valueToCode.size() >= MAX_VARBIN_ENTRIES || dictBytes + added > MAX_VARBIN_DICT_BYTES)) { + if (!first && (valueToCode.size() >= MAX_ENTRIES || dictBytes + added > MAX_DICT_BYTES)) { consumed = i; break; } @@ -360,7 +359,7 @@ private long entryBytes(Object key) { // when it no longer fits. private boolean addVarBinNullEntry() { if (!valueToCode.isEmpty() - && (valueToCode.size() >= MAX_VARBIN_ENTRIES || dictBytes + VIEW_BYTES > MAX_VARBIN_DICT_BYTES)) { + && (valueToCode.size() >= MAX_ENTRIES || dictBytes + VIEW_BYTES > MAX_DICT_BYTES)) { return false; } nullCode = valueToCode.size(); @@ -381,10 +380,9 @@ private void record(short[] codes, boolean[] validity, int len, ProtoScalarValue codeArrayBytes += 2L * len; } - // Gives the null entry the next code, as Rust's dict builder does at the first null; false - // when the dictionary is already full. + // Gives the null entry the next code, as Rust's `encode_null`; false when it no longer fits. private boolean addNullEntry(int cap) { - if (valueToCode.size() == cap) { + if (!valueToCode.isEmpty() && valueToCode.size() >= cap) { return false; } nullCode = valueToCode.size(); @@ -392,23 +390,6 @@ private boolean addNullEntry(int cap) { return true; } - /// Drops every dictionary entry added since the map held `startSize` values, restoring the - /// state a rejected chunk found — the demoting caller replays the already-buffered chunks - /// through [#buildInverseMap] and [#reconstructChunk], which must not see this chunk's values. - private void rollbackTo(int startSize, int startNullCode) { - valueToCode.values().removeIf(code -> code >= startSize); - nullCode = startNullCode; - if (varBin) { - return; - } - bitsIndex = new LongIntMap(INDEX_MIN_CAPACITY); - for (Map.Entry e : valueToCode.entrySet()) { - if (e.getKey() != NULL_KEY) { - bitsIndex.put(valueBits(ptype, e.getKey()), e.getValue()); - } - } - } - /// The chunk's values as raw bit patterns, keyed so that two rows share a pattern exactly when /// their boxed values are `equals` (hence `doubleToLongBits`, which folds every NaN together /// and keeps `-0.0` apart from `0.0`, just like `Double.equals`). I64/U64 arrays are used in @@ -551,57 +532,51 @@ static boolean isVarBinDictCandidate(Object[] data, boolean[] validity) { return seen.size() * 2 < data.length; } - static boolean isDictCandidate(PType ptype, Object data) { - return isDictCandidate(ptype, data, null); + /// Whether the dict layout reads a column of `ptype`: only the carriers the reader's lazy dict + /// decode supports (I32/U32, I64/U64, F64) are admitted. + /// - I8/U8/I16/U16 excluded: dict gives little/no benefit (a U8/U16 code is no smaller than the + /// value), the Rust compressor does not dict them either (verified by + /// RustWritesJavaReadsIntegrationTest#jniWriter_javaReader_lowCardinalityI16), and the reader + /// cannot decode a narrow-int dict -- emitting one produced an unreadable file. + /// - F16/F32 excluded: no measured workload; ALP usually wins. + /// + /// Whether it is worth it is Rust's probe, in the writer: the cascade must pick a dict for the + /// column's first 8192 rows. + /// + /// @param ptype the column's primitive type + /// @return `true` for a type the dict layout reads + static boolean admitsPtype(PType ptype) { + return switch (ptype) { + case I32, U32, I64, U64, F64 -> true; + default -> false; + }; } - /// Like [#isDictCandidate(PType, Object)] but ignores null (invalid) rows when counting distinct - /// values, so a nullable low-cardinality column still qualifies for the shared global dictionary. - /// Null slots hold zero-valued placeholders (per NullableData's contract); skipping them keeps a - /// legitimate `0` value from being deduplicated against those placeholders and keeps nulls out of - /// the distinct count. The ratio denominator stays the total row count (not the valid-row count), - /// matching the per-chunk encoders' convention that a null placeholder occupies a row like any - /// other value. + /// Whether a sample of a primitive column is dictionary material when there is no cascade to ask: + /// Rust's dict scheme skips a sample with more than half of its values distinct, and a single + /// value is a constant, not a dictionary. /// /// @param ptype the column's primitive type - /// @param data the packed values array; positions marked invalid by `validity` are skipped - /// @param validity per-row validity bitmap, or `null` meaning every row is valid - /// @return `true` if the column's distinct valid-value count is low enough to dictionary-encode - static boolean isDictCandidate(PType ptype, Object data, boolean[] validity) { - // Only the carriers the reader's lazy dict decode supports (I32/I64/F64) are admitted. - // - I8/U8/I16/U16 excluded: dict gives little/no benefit (a U8/U16 code is no smaller - // than the value), the Rust compressor does not dict them either (verified by - // RustWritesJavaReadsIntegrationTest#jniWriter_javaReader_lowCardinalityI16), and the - // reader cannot decode a narrow-int dict — emitting one produced an unreadable file. - // - F16/F32 excluded: no measured workload; ALP usually wins. - // F64 admitted: low-card F64 columns (taxi mta_tax/Airport_fee/extra) compress better via - // global dict + sparse-coded codes (matches Rust FloatDictScheme). The skip rule - // (cardinality / 2 below) mirrors Rust's >50%-distinct skip. - if (ptype == PType.I8 || ptype == PType.U8 - || ptype == PType.I16 || ptype == PType.U16 - || ptype == PType.F16 || ptype == PType.F32) { - return false; - } + /// @param data the sample's values; positions marked invalid by `validity` are skipped + /// @param validity per-row validity, or `null` when every row is valid + /// @return `true` if the valid values are under half distinct, and more than one + static boolean isLowCardinality(PType ptype, Object data, boolean[] validity) { int n = primitiveArrayLen(data, ptype); if (n == 0) { return false; } - // Counts distinct values on raw bits rather than into a HashSet, which boxed one - // value per row of the probe chunk — the same cost already removed from ingestDictChunk - // and DictEncodingEncoder, and 8.2% of a bitpacked write. - int distinct = countDistinctCapped(rawBits(data, ptype, n), n, validity); - if (distinct > GLOBAL_DICT_MAX_CARDINALITY || distinct == 0) { - return false; - } - // Single-value columns fit vortex.constant better than dict (zero dict overhead). - // Delegate to the cascading compressor. - if (distinct == 1) { - return false; + long[] bits = rawBits(data, ptype, n); + LongIntMap seen = new LongIntMap(64); + int distinct = 0; + for (int i = 0; i < n; i++) { + if ((validity == null || validity[i]) && seen.add(bits[i])) { + distinct++; + } } - return distinct * 2 < n; + return distinct > 1 && distinct * 2 < n; } - /// Length of a global-dict column's chunk array. Only the dict-admitted carriers ([#isDictCandidate]) + /// Length of a global-dict column's chunk array. Only the dict-admitted carriers ([#admitsPtype]) /// — I32/U32, I64/U64, F64 — reach here; narrow-int and F16/F32 ptypes are rejected upstream. static int primitiveArrayLen(Object data, PType ptype) { return switch (ptype) { @@ -613,7 +588,7 @@ static int primitiveArrayLen(Object data, PType ptype) { } /// Boxed element `i` of a global-dict column's chunk array. Only the dict-admitted carriers - /// ([#isDictCandidate]) — I32/U32, I64/U64, F64 — reach here; narrow-int and F16/F32 ptypes are + /// ([#admitsPtype]) — I32/U32, I64/U64, F64 — reach here; narrow-int and F16/F32 ptypes are /// rejected upstream. static Object readPrimitiveElement(Object data, PType ptype, int i) { return switch (ptype) { @@ -624,31 +599,6 @@ static Object readPrimitiveElement(Object data, PType ptype, int i) { }; } - /// Counts distinct raw bit patterns, abandoning as soon as the count exceeds - /// [#GLOBAL_DICT_MAX_CARDINALITY] — past that the column is not a dictionary candidate, so the - /// exact count is never needed. The set never grows: it is sized once for the cap. - /// - /// @param bits the chunk's values as raw bit patterns - /// @param n element count - /// @param validity per-row validity, or `null` when every row is valid - /// @return the distinct count, or [#GLOBAL_DICT_MAX_CARDINALITY] + 1 if the cap was exceeded - private static int countDistinctCapped(long[] bits, int n, boolean[] validity) { - LongIntMap seen = new LongIntMap(GLOBAL_DICT_MAX_CARDINALITY + 1); - int distinct = 0; - for (int i = 0; i < n; i++) { - if (validity != null && !validity[i]) { - continue; - } - if (seen.add(bits[i])) { - distinct++; - if (distinct > GLOBAL_DICT_MAX_CARDINALITY) { - return distinct; - } - } - } - return distinct; - } - // The dictionary key of one Utf8/Binary row. Utf8 keys are always Strings, a UTF-8 row decoded, // so a column written partly as String[] and partly as byte[][] still codes each value once. private Object dictKey(Object value) { diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java index fa9d00fa2..539c2015c 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java @@ -99,12 +99,12 @@ /// ``` /// /// With global dictionary encoding enabled (the default), candidate columns are buffered in the heap -/// until `close()` so a shared dictionary can span every chunk. Buffering is cardinality-bounded -/// (ADR 0021): each candidate retains a deduplicated value-to-code map capped at -/// `GLOBAL_DICT_MAX_CARDINALITY` plus cheap per-chunk code arrays, so retained memory scales with a -/// column's distinct-value count and row count — not its raw byte size. A column whose distinct set -/// would exceed the cap demotes to per-chunk encoding immediately. [WriteOptions#globalDictMaxRetainedBytes()] -/// (default 1 GB) remains a secondary safety net over the aggregate code-array bytes. +/// until `close()` so a shared dictionary can span every chunk. Buffering is bounded (ADR 0021): each +/// candidate retains a deduplicated value-to-code map of at most 65,535 entries or 1 MiB per +/// dictionary, as Rust's dict layout does, plus cheap per-chunk code arrays; a column that fills a +/// dictionary starts the next, so retained memory scales with the open dictionary and the row count, +/// not the raw byte size. [WriteOptions#globalDictMaxRetainedBytes()] (default 1 GB) remains a +/// secondary safety net over the aggregate code-array bytes. public final class VortexWriter implements Closeable { // Indices into layout_specs list in the FbsFooter @@ -166,12 +166,12 @@ public final class VortexWriter implements Closeable { // Global dict state: columns detected as low-cardinality on their first chunk are buffered here // instead of encoded per-chunk. Buffering is cardinality-bounded (ADR 0021): rather than retain - // raw values, each candidate holds a deduplicated value->code map (capped at - // GLOBAL_DICT_MAX_CARDINALITY) plus cheap per-chunk code arrays. Flushed in close() as one Dict - // layout. When a column's distinct set would exceed the cap mid-file, it demotes immediately. + // raw values, each candidate holds a deduplicated value->code map (at most 65,535 entries or 1 MiB + // per dictionary) plus cheap per-chunk code arrays. Flushed in close() as one Dict layout per + // dictionary. Past the retained-bytes budget the largest columns demote to per-chunk encoding. private final Set dictCandidates = new LinkedHashSet<>(); private final Map dictStates = new LinkedHashMap<>(); - // Dictionaries a text column has already filled to Rust's constraints (a new one starts when it + // Dictionaries a column has already filled to Rust's constraints (a new one starts when it // reaches them); the open one is in dictStates. private final Map> dictRuns = new LinkedHashMap<>(); private final Map> dictColRefs = new LinkedHashMap<>(); @@ -668,7 +668,18 @@ public void writeChunk(Map columns) throws IOException { ? ((io.github.dfa1.vortex.writer.encode.NullableData) data).validity() : null; boolean candidate = false; if (colDtype instanceof DType.Primitive p) { - candidate = DictColumnState.isDictCandidate(p.ptype(), values, validity); + // Rust's DictStrategy admits a column when its probe compressor, the cascade, turns + // the first 8192 rows into a dict; there is no cardinality cap + candidate = DictColumnState.admitsPtype(p.ptype()); + if (candidate && options.allowedCascading() == 0) { + // no cascade to ask: the sample's distinct count decides + int probeRows = (int) Math.min(Repartitioner.BLOCK_ROWS, arrayLength(data)); + Object head = Repartitioner.head(values, probeRows); + candidate = DictColumnState.isLowCardinality(p.ptype(), head, + validity != null ? Arrays.copyOf(validity, probeRows) : null); + } else if (candidate) { + candidate = probeCompressesToDict(colDtype, data); + } } else if (colDtype instanceof DType.Utf8 || colDtype instanceof DType.Binary) { // Rust's dict layout admits Primitive | Utf8 | Binary (dict_layout_supported), by what // its first 8192 rows do: not more than half distinct. The cascade probe stays with @@ -678,9 +689,6 @@ public void writeChunk(Map columns) throws IOException { Arrays.copyOf((Object[]) values, probeRows), validity != null ? Arrays.copyOf(validity, probeRows) : null); } - if (candidate && colDtype instanceof DType.Primitive) { - candidate = probeCompressesToDict(colDtype, data); - } if (candidate) { dictCandidates.add(colName); dictStates.put(colName, new DictColumnState(colDtype)); @@ -688,34 +696,19 @@ public void writeChunk(Map columns) throws IOException { } if (dictCandidates.contains(colName)) { - // Ingest this chunk into the column's cardinality-bounded dict state: dedup values - // into the shared value->code map, buffer a cheap per-chunk code array, and capture - // the per-chunk stats now (before the raw array is discarded). If a new distinct - // value would push the map past GLOBAL_DICT_MAX_CARDINALITY, ingestDictChunk returns - // false and we demote the column immediately (mid-file cap breach, ADR 0021). - boolean admitted; - if (colDtype instanceof DType.Primitive) { - admitted = dictStates.get(colName).ingestDictChunk(data); - } else { - ingestVarBinChunk(colName, colDtype, data); - admitted = true; - } - if (!admitted) { - // Cap breached by this chunk: demote (replaying the already-buffered chunks per - // -chunk) then write this chunk — which ingest rejected without buffering — too. - demoteDictColumn(colName); - appendChunk(colName, colDtype, data); - } else { - long before = dictRetainedBytes.getOrDefault(colName, 0L); - long after = retainedBytes(colName); - dictRetainedTotal += after - before; - dictRetainedBytes.put(colName, after); - if (dictRetainedTotal > dictRetainedBudget) { - // Aggregate code-array budget exceeded (secondary safety net). Demote the - // largest-retained columns until back under budget, so writer memory stays - // bounded even across many wide candidate columns. - evictLargestDictColumnsUntilUnderBudget(); - } + // Ingest this chunk into the column's open dictionary, starting the next one when it is + // full: dedup values into the shared value->code map, buffer a cheap per-chunk code + // array, and capture the per-chunk stats now (before the raw array is discarded). + ingestChunk(colName, colDtype, data); + long before = dictRetainedBytes.getOrDefault(colName, 0L); + long after = retainedBytes(colName); + dictRetainedTotal += after - before; + dictRetainedBytes.put(colName, after); + if (dictRetainedTotal > dictRetainedBudget) { + // Aggregate code-array budget exceeded (secondary safety net). Demote the + // largest-retained columns until back under budget, so writer memory stays + // bounded even across many wide candidate columns. + evictLargestDictColumnsUntilUnderBudget(); } } else { appendChunk(colName, colDtype, data); @@ -1583,15 +1576,15 @@ private static byte[] buildDictLayoutMetaBytes(PType codePType, boolean nullable // orchestration that needs this writer's segment-writing machinery (writeSegment, colChunks, // schema, options). - /// Feeds one text chunk to the column's open dictionary; whatever does not fit under Rust's + /// Feeds one chunk to the column's open dictionary; whatever does not fit under Rust's /// constraints closes it and starts the next one (`DictStrategy`: "when the dict constraints are - /// hit, a new dictionary is created"), so a text column never gives up its dict layout for its + /// hit, a new dictionary is created"), so a column never gives up its dict layout for its /// cardinality. - private void ingestVarBinChunk(ColumnName colName, DType colDtype, Object data) { + private void ingestChunk(ColumnName colName, DType colDtype, Object data) { Object rest = data; while (true) { DictColumnState state = dictStates.get(colName); - int consumed = state.ingestVarBinChunk(rest); + int consumed = colDtype instanceof DType.Primitive ? state.ingestDictChunk(rest) : state.ingestVarBinChunk(rest); long length = arrayLength(rest); if (consumed == length) { return; diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java index d78903347..08d925c8a 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java @@ -12,7 +12,6 @@ import java.util.List; import java.util.stream.Stream; -import static io.github.dfa1.vortex.writer.DictColumnState.GLOBAL_DICT_MAX_CARDINALITY; import static org.assertj.core.api.Assertions.assertThat; import static org.junit.jupiter.params.provider.Arguments.arguments; @@ -22,73 +21,24 @@ /// fixes each cardinality / ratio edge. class DictColumnStateTest { - // ── isDictCandidate (primitive) ────────────────────────────────────────────── + // ── admitsPtype ────────────────────────────────────────────────────────────── - static Stream dictCandidateCases() { + static Stream admittedPtypes() { return Stream.of( // exclusions: a U8/U16 code is no smaller than the value, and F16/F32 prefer ALP - arguments("F32 excluded", PType.F32, new float[]{1f, 1f, 2f, 2f, 2f}, false), - arguments("F16 excluded", PType.F16, new short[]{1, 1, 2, 2, 2}, false), - arguments("I8 excluded", PType.I8, new byte[]{1, 2, 1, 2, 1}, false), - arguments("U8 excluded", PType.U8, new byte[]{1, 2, 1, 2, 1}, false), - arguments("I16 excluded", PType.I16, new short[]{1, 2, 1, 2, 1}, false), - arguments("U16 excluded", PType.U16, new short[]{1, 2, 1, 2, 1}, false), - // admitted carriers, 2 distinct over 5 rows (4 < 5, under the gate) - arguments("I64 low cardinality", PType.I64, new long[]{1, 2, 1, 2, 1}, true), - arguments("F64 low cardinality", PType.F64, new double[]{1, 2, 1, 2, 1}, true), - arguments("empty", PType.I64, new long[0], false), - arguments("single distinct value", PType.I64, new long[]{7, 7, 7, 7}, false), - arguments("ratio exactly 50%", PType.I64, new long[]{1, 2, 1, 2}, false), - arguments("ratio under 50%", PType.I64, new long[]{1, 2, 1, 2, 1}, true), - arguments("cardinality at MAX", PType.I64, distinctThenRepeat(GLOBAL_DICT_MAX_CARDINALITY), true), - arguments("cardinality over MAX", PType.I64, distinctThenRepeat(GLOBAL_DICT_MAX_CARDINALITY + 1), false)); + arguments(PType.F32, false), arguments(PType.F16, false), arguments(PType.I8, false), + arguments(PType.U8, false), arguments(PType.I16, false), arguments(PType.U16, false), + arguments(PType.I32, true), arguments(PType.U32, true), arguments(PType.I64, true), + arguments(PType.U64, true), arguments(PType.F64, true)); } @ParameterizedTest(name = "{0}") - @MethodSource("dictCandidateCases") - void isDictCandidate(String name, PType ptype, Object data, boolean expected) { - // Given — a column of `ptype` with the case's data + @MethodSource("admittedPtypes") + void admitsPtype(PType ptype, boolean expected) { + // Given — a column of `ptype` // When - boolean result = DictColumnState.isDictCandidate(ptype, data); - - // Then - assertThat(result).isEqualTo(expected); - } - - // ── isDictCandidate (primitive, nullable) ──────────────────────────────────── - - static Stream nullableDictCandidateCases() { - // The values array holds a zero placeholder at every null slot (NullableData contract). Nulls - // must not count toward cardinality; the ratio denominator stays the total row count. - return Stream.of( - // 2 distinct valid values (1, 2) over 5 total rows: 2*2 < 5 → candidate. The null slot - // (index 2, placeholder 0) is skipped, so 0 does not inflate cardinality to 3. - arguments("nulls excluded from cardinality", - new long[]{1, 2, 0, 1, 2}, new boolean[]{true, true, false, true, true}, true), - // All-null: no valid values, so never a candidate (nothing to dictionary). - arguments("all-null not a candidate", - new long[]{0, 0, 0}, new boolean[]{false, false, false}, false), - // Null-heavy but only 2 distinct valid values over 8 total rows (2*2 < 8) → candidate. - arguments("null-heavy but low cardinality still passes", - new long[]{1, 0, 0, 2, 0, 1, 0, 2}, - new boolean[]{true, false, false, true, false, true, false, true}, true), - // Placeholder 0 collides with a legitimate valid 0: skipping nulls keeps 0 as a real - // distinct value only from its valid occurrence (2 distinct: 0 and 5). - arguments("placeholder zero not deduped against valid zero", - new long[]{0, 0, 5, 0, 5}, new boolean[]{true, false, true, false, true}, true), - // validity == null delegates to the all-valid path: single distinct value → not a - // candidate (constant encoding wins). - arguments("null validity single value", new long[]{7, 7, 7, 7}, null, false)); - } - - @ParameterizedTest(name = "{0}") - @MethodSource("nullableDictCandidateCases") - void isDictCandidate_nullable(String name, long[] data, boolean[] validity, boolean expected) { - // Given — an I64 column with the case's values and validity - - // When - boolean result = DictColumnState.isDictCandidate(PType.I64, data, validity); + boolean result = DictColumnState.admitsPtype(ptype); // Then assertThat(result).isEqualTo(expected); @@ -103,7 +53,7 @@ static Stream utf8DictCandidateCases() { arguments("ratio under 50%", new String[]{"a", "b", "a", "b", "a"}, true), // no cardinality cap as for numbers: Rust skips a sample only above half distinct, and a // column that outgrows one dictionary starts another - arguments("Utf8 admitted above numeric cap", repeated(GLOBAL_DICT_MAX_CARDINALITY + 1), true), + arguments("Utf8 admitted above numeric cap", repeated(2_049), true), arguments("cardinality of a full dictionary", repeated(70_000), true)); } @@ -201,22 +151,21 @@ void ingestDictChunk_assignsCodesFirstSeen_nullIncluded() { boolean[] validity = {true, true, false, true, false, true}; // When - boolean admitted = sut.ingestDictChunk(new NullableData(values, validity)); + int admitted = sut.ingestDictChunk(new NullableData(values, validity)); short[] result = sut.chunkCodes(0); // Then - assertThat(admitted).isTrue(); + assertThat(admitted).isEqualTo(values.length); assertThat(result).containsExactly((short) 0, (short) 1, (short) 2, (short) 1, (short) 2, (short) 3); assertThat(sut.nullCode()).isEqualTo(2); assertThat((double[]) sut.uniques()).containsExactly(1.75, 0.0, 0.0, -1.75); } @Test - void ingestDictChunk_rejectedChunk_rollsBackItsNullEntry() { - // Given — a full dictionary; the next chunk brings its first null and then a new value, so - // the chunk is rejected and must not leave a null entry behind for the demotion replay + void ingestDictChunk_stopsAtTheNullEntryThatNoLongerFits() { + // Given — a dictionary filled to u16::MAX entries; the next chunk's first row is a null var sut = new DictColumnState(new DType.Primitive(PType.I64, true)); - long[] full = new long[DictColumnState.GLOBAL_DICT_MAX_CARDINALITY - 1]; + long[] full = new long[DictColumnState.MAX_ENTRIES]; for (int i = 0; i < full.length; i++) { full[i] = i; } @@ -225,14 +174,34 @@ void ingestDictChunk_rejectedChunk_rollsBackItsNullEntry() { boolean[] validity = {false, true}; // When - boolean result = sut.ingestDictChunk(new NullableData(next, validity)); + int result = sut.ingestDictChunk(new NullableData(next, validity)); - // Then - assertThat(result).isFalse(); + // Then — Rust's encode_null: no room for a null entry, so the dictionary is closed + assertThat(result).isZero(); assertThat(sut.nullCode()).isEqualTo(-1); assertThat(sut.cardinality()).isEqualTo(full.length); } + @Test + void ingestDictChunk_stopsAtTheValueThatNoLongerFitsAndKeepsThePrefixStats() { + // Given — a dictionary filled to u16::MAX entries + var sut = new DictColumnState(new DType.Primitive(PType.I64, false)); + long[] full = new long[DictColumnState.MAX_ENTRIES]; + for (int i = 0; i < full.length; i++) { + full[i] = i; + } + sut.ingestDictChunk(full); + + // When — known values still fit; the first new one ends the prefix + int result = sut.ingestDictChunk(new long[]{5, 6, -1, 7}); + + // Then — and the prefix's own bounds are recorded, not the whole chunk's + assertThat(result).isEqualTo(2); + assertThat(sut.chunkRowCounts()).containsExactly((long) full.length, 2L); + assertThat(sut.chunkStatsMin().getLast()).isEqualTo(ProtoScalarValue.ofInt64Value(5L)); + assertThat(sut.chunkStatsMax().getLast()).isEqualTo(ProtoScalarValue.ofInt64Value(6L)); + } + // ── primitiveArrayLen / readPrimitiveElement ───────────────────────────────── static Stream primitiveArrayLenCases() { @@ -335,7 +304,7 @@ private static String[] repeated(int distinct) { void ingestVarBinChunk_stopsAtTheEntryThatNoLongerFits() { // Given — a dictionary filled to u16::MAX entries DictColumnState sut = new DictColumnState(new DType.Utf8(false)); - String[] full = new String[DictColumnState.MAX_VARBIN_ENTRIES]; + String[] full = new String[DictColumnState.MAX_ENTRIES]; for (int i = 0; i < full.length; i++) { full[i] = "v" + i; } @@ -346,7 +315,7 @@ void ingestVarBinChunk_stopsAtTheEntryThatNoLongerFits() { // Then assertThat(consumed).isEqualTo(2); - assertThat(sut.cardinality()).isEqualTo(DictColumnState.MAX_VARBIN_ENTRIES); + assertThat(sut.cardinality()).isEqualTo(DictColumnState.MAX_ENTRIES); assertThat(sut.chunkRowCounts()).containsExactly((long) full.length, 2L); } @@ -363,14 +332,14 @@ void ingestVarBinChunk_countsViewsAndOutOfLineBytesAgainstTheByteLimit() { int consumed = sut.ingestVarBinChunk(values); // Then - assertThat(consumed).isEqualTo((int) (DictColumnState.MAX_VARBIN_DICT_BYTES / 36)); + assertThat(consumed).isEqualTo((int) (DictColumnState.MAX_DICT_BYTES / 36)); } @Test void ingestVarBinChunk_admitsTheFirstEntryWhateverItsSize() { // Given — one value past the byte limit on its own DictColumnState sut = new DictColumnState(new DType.Utf8(false)); - String huge = "x".repeat((int) DictColumnState.MAX_VARBIN_DICT_BYTES + 1); + String huge = "x".repeat((int) DictColumnState.MAX_DICT_BYTES + 1); // When int consumed = sut.ingestVarBinChunk(new String[]{huge, "other"}); @@ -383,7 +352,7 @@ void ingestVarBinChunk_admitsTheFirstEntryWhateverItsSize() { void ingestVarBinChunk_aFullDictionaryTakesNothingNew() { // Given DictColumnState sut = new DictColumnState(new DType.Utf8(false)); - sut.ingestVarBinChunk(new String[]{"x".repeat((int) DictColumnState.MAX_VARBIN_DICT_BYTES + 1)}); + sut.ingestVarBinChunk(new String[]{"x".repeat((int) DictColumnState.MAX_DICT_BYTES + 1)}); // When int consumed = sut.ingestVarBinChunk(new String[]{"new"}); diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java index 0c2c4b067..564beb772 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java @@ -153,15 +153,10 @@ void cardinalityExceededAcrossChunks_i64_fallsBackToPerChunk(@TempDir Path tmp) } @Test - void cardinalityGrowsPastCapMidFile_i64_demotesImmediatelyToChunkedLayout(@TempDir Path tmp) throws IOException { - // Given — the core ADR 0021 case. The first chunk is a genuine low-cardinality dict candidate - // (200 distinct / 1000 rows), so the column is admitted and buffered as codes. Later chunks - // introduce fresh uniques until the distinct set crosses GLOBAL_DICT_MAX_CARDINALITY (2048) - // partway through the file. The OLD raw-buffering path would have accumulated every raw value - // for the whole file before catching this at close(); the new path detects the breach at - // ingest and demotes IMMEDIATELY. We can't assert on memory in a unit test, but the OUTPUT - // SHAPE proves the demotion happened: a Chunked-of-Flats layout (one Flat per chunk), NOT a - // single shared Dict layout. A Dict here would mean the whole column was still buffered. + void cardinalityGrowsPastTheOldCapMidFile_i64_staysOneDictionary(@TempDir Path tmp) throws IOException { + // Given — the first chunk is a genuine low-cardinality dict candidate (200 distinct / 1000 + // rows). Later chunks introduce fresh uniques until the distinct set crosses 2048, the cap + // the writer used to demote at. Rust has no such cap: its dictionary holds up to 65,535 values. var schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(DTypes.I64), false); Path file = tmp.resolve("card_grows_i64.vortex"); int chunkCount = 6; @@ -170,8 +165,7 @@ void cardinalityGrowsPastCapMidFile_i64_demotesImmediatelyToChunkedLayout(@TempD long next = 0; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, WriteOptions.cascading(3))) { - // When — chunk 0 is low-card (200 distinct), later chunks are all-unique so the running - // distinct count climbs ~1000/chunk and passes 2048 during chunk 2 or 3. + // When for (int c = 0; c < chunkCount; c++) { long[] data = new long[rowsPerChunk]; for (int i = 0; i < rowsPerChunk; i++) { @@ -183,16 +177,41 @@ void cardinalityGrowsPastCapMidFile_i64_demotesImmediatelyToChunkedLayout(@TempD } } - // Then — the demoted column is written per chunk, never as a shared Dict. + // Then — still one shared Dict layout, and every value round-trips try (var vf = VortexReader.open(file, ReadRegistry.loadAll())) { var columnLayout = unwrapZoned(vf.layout().children().getFirst()); - assertThat(columnLayout.isDict()).as("cardinality-demoted column must not be a global dict").isFalse(); - // The replayed batches coalesce like any column's (Rust's repartition, #470): well - // under 1 MB, so one Flat. - assertThat(columnLayout.children()).as("demoted rows coalesce into one chunk") - .singleElement().satisfies(child -> assertThat(child.isFlat()).isTrue()); + assertThat(columnLayout.isDict()).as("one dictionary holds 5,200 values").isTrue(); + assertThat(readAllLongs(vf, "v")).containsExactly(expected); + } + } - // And every value round-trips exactly across all chunks despite the mid-file demotion. + @Test + void cardinalityPastAFullDictionary_i64_startsANewOne(@TempDir Path tmp) throws IOException { + // Given — the first 8192 rows hold 2,000 distinct values, so the column is admitted; every row after + // them is new, which takes the dictionary past 65,535 values + var schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(DTypes.I64), false); + Path file = tmp.resolve("runs_i64.vortex"); + int chunkCount = 8; + int rowsPerChunk = 40_000; + long[] expected = new long[chunkCount * rowsPerChunk]; + try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var sut = VortexWriter.create(ch, schema, WriteOptions.cascading(3))) { + // When + for (int c = 0; c < chunkCount; c++) { + long[] data = new long[rowsPerChunk]; + for (int i = 0; i < rowsPerChunk; i++) { + data[i] = c == 0 && i < 8_192 ? wide(i % 2_000) : wide(100_000L + (long) c * rowsPerChunk + i); + expected[c * rowsPerChunk + i] = data[i]; + } + sut.writeChunk(Map.of(ColumnName.of("v"), data)); + } + } + + // Then — a chunked layout of dict layouts, and every value round-trips + try (var vf = VortexReader.open(file, ReadRegistry.loadAll())) { + var columnLayout = unwrapZoned(vf.layout().children().getFirst()); + assertThat(columnLayout.isChunked()).isTrue(); + assertThat(columnLayout.children()).hasSizeGreaterThan(1).allSatisfy(child -> assertThat(child.isDict()).isTrue()); assertThat(readAllLongs(vf, "v")).containsExactly(expected); } }