diff --git a/CHANGELOG.md b/CHANGELOG.md index 319f2b715..8db2dc510 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - FSST symbol tables are trained as Rust's `fsst-rs` does (its sample, generations, candidate gains and cut), so a text column without a null gets Rust's FSST encoding up to ties of equal gain, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - `VortexReader.columnStats()` reports the min and max of a decimal column, and a decimal filter compares by value instead of by the long it truncates to. ([#583](https://github.com/dfa1/vortex-java/issues/583)) - A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650)) +- 8-bit, 16-bit and `F32` columns get Rust's dict layout when its probe picks one, so a nullable low-cardinality `SMALLINT` is written up to 30% smaller. ([#651](https://github.com/dfa1/vortex-java/issues/651)) - A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696)) - **Breaking:** a `Double`, `Float` or `BigDecimal` filter operand on an integer column fails with `VortexException`, as vortex-jni does, instead of being truncated so that `x >= 1.5` matched `x = 1`. ([#701](https://github.com/dfa1/vortex-java/issues/701)) - A `vortex.variant` column whose shredded dtype has a control character in a field or variant name fails with `VortexException` instead of `IllegalArgumentException`. ([#694](https://github.com/dfa1/vortex-java/issues/694)) diff --git a/docs/compatibility.md b/docs/compatibility.md index 4942f5149..39966ce58 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -129,7 +129,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul | `vortex.varbinview` | `VarBinViewEncodingDecoder` | `VarBinViewEncodingEncoder` | ✅ | ✅ | Utf8, Binary | | `vortex.alp` | `AlpEncodingDecoder` | `AlpEncodingEncoder` | ✅ | ✅ | F64, F32 | | `vortex.alprd` | `AlpRdEncodingDecoder` | `AlpRdEncodingEncoder` | ✅ | ✅ | F64, F32 | -| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and Rust's probe: the cascade, run on the column's first 8,192 rows, must pick `vortex.dict` (Rust's `DictStrategy`). Text columns skip the probe, as our cascade picks OnPair where Rust's string dictionary wins, and need under half of those rows distinct. Neither has a cardinality cap: whenever the open dictionary reaches 65,535 entries or 1 MiB the writer closes it and starts the next, and a column of several dictionaries is a chunked layout of dict layouts, as in Rust. With `cascading(0)` there is no cascade to probe, so the distinct count of those rows (under half, and more than one) decides. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust | +| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. Any primitive column can qualify, as in Rust's `dict_layout_supported`, by Rust's probe: the cascade, run on the column's first 8,192 rows, must pick `vortex.dict` (Rust's `DictStrategy`). Text columns skip the probe, as our cascade picks OnPair where Rust's string dictionary wins, and need under half of those rows distinct. Neither has a cardinality cap: whenever the open dictionary reaches 65,535 entries or 1 MiB the writer closes it and starts the next, and a column of several dictionaries is a chunked layout of dict layouts, as in Rust. With `cascading(0)` there is no cascade to probe, so the distinct count of those rows (under half, and more than one) decides. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust | | `vortex.sparse` | `SparseEncodingDecoder` | `SparseEncodingEncoder` | ✅ | ✅ | Primitive, Bool, Utf8/Binary (decode); Primitive (encode) | | `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) | | `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct | diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java index c7257126e..5b5503337 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/DictLayoutInteropIntegrationTest.java @@ -1,5 +1,9 @@ package io.github.dfa1.vortex.integration; +import dev.vortex.api.DataSource; +import dev.vortex.api.Expression; +import dev.vortex.api.Partition; +import dev.vortex.api.Scan; import dev.vortex.api.Session; import dev.vortex.api.VortexWriter; import dev.vortex.arrow.ArrowAllocation; @@ -7,17 +11,24 @@ import io.github.dfa1.vortex.core.model.ColumnName; import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.core.model.PType; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.ScanOptions; import io.github.dfa1.vortex.reader.VortexReader; import io.github.dfa1.vortex.reader.array.VarBinArray; import io.github.dfa1.vortex.writer.WriteOptions; +import io.github.dfa1.vortex.writer.encode.NullableData; import org.apache.arrow.c.ArrowArray; import org.apache.arrow.c.ArrowSchema; import org.apache.arrow.c.Data; import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.FieldVector; +import org.apache.arrow.vector.SmallIntVector; +import org.apache.arrow.vector.TinyIntVector; +import org.apache.arrow.vector.UInt2Vector; import org.apache.arrow.vector.VarCharVector; import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; import org.apache.arrow.vector.types.pojo.ArrowType; import org.apache.arrow.vector.types.pojo.Field; import org.apache.arrow.vector.types.pojo.Schema; @@ -25,6 +36,7 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; import org.junit.jupiter.params.provider.ValueSource; import java.io.IOException; @@ -142,6 +154,131 @@ void startsANewNumericDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws E assertThat(readAllLongs(java)).containsExactly(values); } + @ParameterizedTest(name = "{0}") + @EnumSource(value = PType.class, names = {"I8", "I16", "U16"}) + void admitsANullableNarrowColumnAsRustDoes(PType ptype, @TempDir Path tmp) throws Exception { + // Given — the profile of Public BI Generico's `Disco` first 8192 rows (#651): a nullable + // SMALLINT, 40% valid, 62 distinct values in 5..132 with a few far more common than the + // rest, value runs of about 2 rows between null runs of about 12. Rust's dict layout admits + // every primitive and writes it as one values pool plus run-end codes; Java filtered narrow + // widths out before the probe and wrote a 1.44x larger bit-packed column with a validity + // per chunk + Random r = new Random(651); + int[] pool = new int[62]; + for (int p = 0; p < pool.length; p++) { + pool[p] = 5 + p * 2; + } + int n = CHUNK * 3; + int[] values = new int[n]; + boolean[] valid = new boolean[n]; + int i = 0; + while (i < n) { + boolean isValid = r.nextInt(100) >= 22; + int run = isValid ? 1 + r.nextInt(3) : 1 + r.nextInt(23); + int value = pool[Math.min(r.nextInt(pool.length), r.nextInt(pool.length))]; + for (int j = 0; j < run && i < n; j++, i++) { + values[i] = value; + valid[i] = isValid; + } + } + + // When + Path java = writeJavaNullable(tmp, ptype, values, valid); + Path jni = writeJniNullable(tmp, ptype, values, valid); + + // Then — the same layouts and encodings, node for node, and vortex-jni reads Java's file + // back to the same values as its own + assertThat(EncodingTrees.of(java)).isEqualTo(EncodingTrees.of(jni)); + assertThat(dictLayouts(EncodingTrees.of(java))).isPositive(); + assertThat(readWithJni(java)).isEqualTo(readWithJni(jni)); + } + + private static Path writeJavaNullable(Path tmp, PType ptype, int[] values, boolean[] valid) throws IOException { + Path file = tmp.resolve("java.vortex"); + ColumnName col = ColumnName.of("v"); + DType.Struct schema = DType.structBuilder().field(col, new DType.Primitive(ptype, true)).build(); + try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var w = io.github.dfa1.vortex.writer.VortexWriter.create(ch, schema, WriteOptions.defaults())) { + for (int from = 0; from < values.length; from += CHUNK) { + Object chunk = switch (ptype) { + case I8 -> { + byte[] a = new byte[CHUNK]; + for (int i = 0; i < CHUNK; i++) { + a[i] = (byte) values[from + i]; + } + yield a; + } + default -> { + short[] a = new short[CHUNK]; + for (int i = 0; i < CHUNK; i++) { + a[i] = (short) values[from + i]; + } + yield a; + } + }; + w.writeChunk(Map.of(col, new NullableData(chunk, java.util.Arrays.copyOfRange(valid, from, from + CHUNK)))); + } + } + return file; + } + + private static Path writeJniNullable(Path tmp, PType ptype, int[] values, boolean[] valid) throws Exception { + Path file = tmp.resolve("jni.vortex"); + ArrowType type = switch (ptype) { + case I8 -> new ArrowType.Int(8, true); + case U16 -> new ArrowType.Int(16, false); + default -> new ArrowType.Int(16, true); + }; + Schema arrow = new Schema(List.of(Field.nullable("v", type))); + try (VortexWriter w = VortexWriter.builder(SESSION, file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) { + for (int from = 0; from < values.length; from += CHUNK) { + try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) { + FieldVector v = root.getVector("v"); + root.allocateNew(); + for (int i = 0; i < CHUNK; i++) { + int at = from + i; + if (!valid[at]) { + v.setNull(i); + continue; + } + switch (v) { + case TinyIntVector t -> t.setSafe(i, values[at]); + case UInt2Vector u -> u.setSafe(i, values[at]); + case SmallIntVector s -> s.setSafe(i, values[at]); + default -> throw new IllegalStateException(v.getClass().getName()); + } + } + root.setRowCount(CHUNK); + try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema as = ArrowSchema.allocateNew(ALLOCATOR)) { + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, as); + w.writeBatch(array.memoryAddress(), as.memoryAddress()); + } + } + } + } + return file; + } + + /// Every row of column `v` as vortex-jni reads it, `null` for a null row. + private static List readWithJni(Path file) throws Exception { + List out = new ArrayList<>(); + DataSource ds = DataSource.open(SESSION, file.toAbsolutePath().toUri().toString()); + Scan scan = ds.scan(dev.vortex.api.ScanOptions.builder() + .projection(Expression.select(new String[]{"v"}, Expression.root())).build()); + while (scan.hasNext()) { + Partition partition = scan.next(); + try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) { + while (reader.loadNextBatch()) { + FieldVector vec = reader.getVectorSchemaRoot().getVector("v"); + for (int i = 0; i < vec.getValueCount(); i++) { + out.add(vec.getObject(i)); + } + } + } + } + return out; + } + private static Path writeJava(Path tmp, long[] values) throws IOException { Path file = tmp.resolve("java.vortex"); DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.I64), false); diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java b/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java index 72413c234..3ec25bee8 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/DictColumnState.java @@ -169,36 +169,14 @@ Object uniques() { } return out; } - return switch (ptype) { - case I32, U32 -> { - int[] a = new int[n]; - for (int i = 0; i < n; i++) { - if (i != nullCode) { - a[i] = (Integer) inverse[i]; - } - } - yield a; - } - case I64, U64 -> { - long[] a = new long[n]; - for (int i = 0; i < n; i++) { - if (i != nullCode) { - a[i] = (Long) inverse[i]; - } - } - yield a; - } - case F64 -> { - double[] a = new double[n]; - for (int i = 0; i < n; i++) { - if (i != nullCode) { - a[i] = (Double) inverse[i]; - } - } - yield a; - } - default -> throw new IllegalStateException("ptype not admitted to the global dict: " + ptype); - }; + // every code but the null entry's names a value; its slot keeps the carrier's zero + short[] codes = new short[n]; + boolean[] valid = new boolean[n]; + for (int i = 0; i < n; i++) { + codes[i] = (short) i; + valid[i] = i != nullCode; + } + return reconstructPrimitiveValues(ptype, codes, valid, inverse); } /// Ingests the leading rows of a primitive chunk that fit this dictionary under Rust's constraints @@ -262,8 +240,11 @@ int ingestDictChunk(Object data) { private static Object prefix(Object values, int n) { return switch (values) { + case byte[] a -> Arrays.copyOf(a, n); + case short[] a -> Arrays.copyOf(a, n); case int[] a -> Arrays.copyOf(a, n); case long[] a -> Arrays.copyOf(a, n); + case float[] a -> Arrays.copyOf(a, n); case double[] a -> Arrays.copyOf(a, n); default -> throw new IllegalStateException("not a global dict carrier: " + values.getClass()); }; @@ -401,42 +382,48 @@ private boolean addNullEntry(int cap) { } /// The chunk's values as raw bit patterns, keyed so that two rows share a pattern exactly when - /// their boxed values are `equals` (hence `doubleToLongBits`, which folds every NaN together - /// and keeps `-0.0` apart from `0.0`, just like `Double.equals`). I64/U64 arrays are used in - /// place; only the narrower carriers pay for a widened copy. + /// their boxed values are `equals` (hence `doubleToLongBits` / `floatToIntBits`, which fold every + /// NaN together and keep `-0.0` apart from `0.0`, just like `Double.equals`). I64/U64 arrays are + /// used in place; only the narrower carriers pay for a widened copy. F16 keys on its raw bits. private static long[] rawBits(Object values, PType ptype, int len) { + if (ptype == PType.I64 || ptype == PType.U64) { + return (long[]) values; + } + long[] out = new long[len]; switch (ptype) { - case I64, U64 -> { - return (long[]) values; + case I8, U8 -> { + byte[] src = (byte[]) values; + for (int i = 0; i < len; i++) { + out[i] = src[i]; + } + } + case I16, U16, F16 -> { + short[] src = (short[]) values; + for (int i = 0; i < len; i++) { + out[i] = src[i]; + } } case I32, U32 -> { int[] src = (int[]) values; - long[] out = new long[len]; for (int i = 0; i < len; i++) { out[i] = src[i]; } - return out; + } + case F32 -> { + float[] src = (float[]) values; + for (int i = 0; i < len; i++) { + out[i] = Float.floatToIntBits(src[i]); + } } case F64 -> { double[] src = (double[]) values; - long[] out = new long[len]; for (int i = 0; i < len; i++) { out[i] = Double.doubleToLongBits(src[i]); } - return out; } - default -> throw new IllegalStateException("ptype not admitted to the global dict: " + ptype); + default -> throw new IllegalStateException("unreachable: " + ptype); } - } - - /// Inverse of [#rawBits] for a single already-boxed dictionary key. - private static long valueBits(PType ptype, Object value) { - return switch (ptype) { - case I32, U32 -> (Integer) value; - case I64, U64 -> (Long) value; - case F64 -> Double.doubleToLongBits((Double) value); - default -> throw new IllegalStateException("ptype not admitted to the global dict: " + ptype); - }; + return out; } /// Inverse of this column's first-seen value-to-code map: `inverse[code]` is the value with that @@ -478,6 +465,24 @@ private static Object reconstructVarBinValues(boolean binary, short[] codes, boo private static Object reconstructPrimitiveValues(PType ptype, short[] codes, boolean[] validity, Object[] inverse) { int len = codes.length; return switch (ptype) { + case I8, U8 -> { + byte[] arr = new byte[len]; + for (int i = 0; i < len; i++) { + if (validity == null || validity[i]) { + arr[i] = (Byte) inverse[codes[i] & 0xFFFF]; + } + } + yield arr; + } + case I16, U16, F16 -> { + short[] arr = new short[len]; + for (int i = 0; i < len; i++) { + if (validity == null || validity[i]) { + arr[i] = (Short) inverse[codes[i] & 0xFFFF]; + } + } + yield arr; + } case I32, U32 -> { int[] arr = new int[len]; for (int i = 0; i < len; i++) { @@ -496,6 +501,15 @@ private static Object reconstructPrimitiveValues(PType ptype, short[] codes, boo } yield arr; } + case F32 -> { + float[] arr = new float[len]; + for (int i = 0; i < len; i++) { + if (validity == null || validity[i]) { + arr[i] = (Float) inverse[codes[i] & 0xFFFF]; + } + } + yield arr; + } case F64 -> { double[] arr = new double[len]; for (int i = 0; i < len; i++) { @@ -505,7 +519,6 @@ private static Object reconstructPrimitiveValues(PType ptype, short[] codes, boo } yield arr; } - default -> throw new IllegalStateException("ptype not admitted to the global dict: " + ptype); }; } @@ -542,25 +555,6 @@ static boolean isVarBinDictCandidate(Object[] data, boolean[] validity) { return seen.size() * 2 < data.length; } - /// Whether the dict layout reads a column of `ptype`: only the carriers the reader's lazy dict - /// decode supports (I32/U32, I64/U64, F64) are admitted. - /// - I8/U8/I16/U16 excluded: dict gives little/no benefit (a U8/U16 code is no smaller than the - /// value), the Rust compressor does not dict them either (verified by - /// RustWritesJavaReadsIntegrationTest#jniWriter_javaReader_lowCardinalityI16), and the reader - /// cannot decode a narrow-int dict -- emitting one produced an unreadable file. - /// - F16/F32 excluded: no measured workload; ALP usually wins. - /// - /// Whether it is worth it is Rust's probe, in the writer: the cascade must pick a dict for the - /// column's first 8192 rows. - /// - /// @param ptype the column's primitive type - /// @return `true` for a type the dict layout reads - static boolean admitsPtype(PType ptype) { - return switch (ptype) { - case I32, U32, I64, U64, F64 -> true; - default -> false; - }; - } /// Whether a sample of a primitive column is dictionary material when there is no cascade to ask: /// Rust's dict scheme skips a sample with more than half of its values distinct, and a single @@ -586,26 +580,28 @@ static boolean isLowCardinality(PType ptype, Object data, boolean[] validity) { return distinct > 1 && distinct * 2 < n; } - /// Length of a global-dict column's chunk array. Only the dict-admitted carriers ([#admitsPtype]) - /// — I32/U32, I64/U64, F64 — reach here; narrow-int and F16/F32 ptypes are rejected upstream. + /// Length of a global-dict column's chunk array, the carrier of its `ptype` (F16 as its raw + /// `short` bits). static int primitiveArrayLen(Object data, PType ptype) { return switch (ptype) { + case I8, U8 -> ((byte[]) data).length; + case I16, U16, F16 -> ((short[]) data).length; case I32, U32 -> ((int[]) data).length; case I64, U64 -> ((long[]) data).length; + case F32 -> ((float[]) data).length; case F64 -> ((double[]) data).length; - default -> throw new IllegalStateException("ptype not admitted to the global dict: " + ptype); }; } - /// Boxed element `i` of a global-dict column's chunk array. Only the dict-admitted carriers - /// ([#admitsPtype]) — I32/U32, I64/U64, F64 — reach here; narrow-int and F16/F32 ptypes are - /// rejected upstream. + /// Boxed element `i` of a global-dict column's chunk array, the carrier of its `ptype`. static Object readPrimitiveElement(Object data, PType ptype, int i) { return switch (ptype) { + case I8, U8 -> ((byte[]) data)[i]; + case I16, U16, F16 -> ((short[]) data)[i]; case I32, U32 -> ((int[]) data)[i]; case I64, U64 -> ((long[]) data)[i]; + case F32 -> ((float[]) data)[i]; case F64 -> ((double[]) data)[i]; - default -> throw new IllegalStateException("ptype not admitted to the global dict: " + ptype); }; } diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/GlobalDictWriter.java b/writer/src/main/java/io/github/dfa1/vortex/writer/GlobalDictWriter.java index 451018eee..efba4662e 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/GlobalDictWriter.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/GlobalDictWriter.java @@ -85,16 +85,16 @@ void detect(ColumnName colName, DType colDtype, Object data) { } boolean candidate = false; if (colDtype instanceof DType.Primitive p) { - // Rust's DictStrategy admits a column when its probe compressor, the cascade, turns - // the first 8192 rows into a dict; there is no cardinality cap - candidate = DictColumnState.admitsPtype(p.ptype()); - if (candidate && options.allowedCascading() == 0) { + // Rust's DictStrategy admits every primitive (dict_layout_supported) when its probe + // compressor, the cascade, turns the first 8192 rows into a dict; there is no + // cardinality cap and no width filter (#651: a nullable I16 was never probed) + if (options.allowedCascading() == 0) { // no cascade to ask: the sample's distinct count decides int probeRows = Math.min(Repartitioner.BLOCK_ROWS, RowCount.of(data)); Object head = Repartitioner.head(values, probeRows); candidate = DictColumnState.isLowCardinality(p.ptype(), head, validity != null ? Arrays.copyOf(validity, probeRows) : null); - } else if (candidate) { + } else { candidate = encoder.probeCompressesToDict(colName, colDtype, data); } } else if (colDtype instanceof DType.Utf8 || colDtype instanceof DType.Binary) { diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java index 08d925c8a..553a10aed 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/DictColumnStateTest.java @@ -7,6 +7,7 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.EnumSource; import org.junit.jupiter.params.provider.MethodSource; import java.util.List; @@ -21,27 +22,35 @@ /// fixes each cardinality / ratio edge. class DictColumnStateTest { - // ── admitsPtype ────────────────────────────────────────────────────────────── - - static Stream admittedPtypes() { - return Stream.of( - // exclusions: a U8/U16 code is no smaller than the value, and F16/F32 prefer ALP - arguments(PType.F32, false), arguments(PType.F16, false), arguments(PType.I8, false), - arguments(PType.U8, false), arguments(PType.I16, false), arguments(PType.U16, false), - arguments(PType.I32, true), arguments(PType.U32, true), arguments(PType.I64, true), - arguments(PType.U64, true), arguments(PType.F64, true)); - } + // ── every primitive ptype ────────────────────────────────────────────────────── @ParameterizedTest(name = "{0}") - @MethodSource("admittedPtypes") - void admitsPtype(PType ptype, boolean expected) { - // Given — a column of `ptype` + @EnumSource(PType.class) + void ingestThenReconstruct_everyPtype_roundTripsInItsCarrier(PType ptype) { + // Given — a nullable chunk of 3 distinct values and a null in the ptype's own carrier. Rust's + // dict layout admits every primitive (#651: a nullable I16 was never probed), so each carrier + // must survive ingest, the dictionary entries, and a demotion's reconstruction + Object values = switch (ptype) { + case I8, U8 -> new byte[]{7, 8, 0, 7, 9}; + case I16, U16, F16 -> new short[]{7, 8, 0, 7, 9}; + case I32, U32 -> new int[]{7, 8, 0, 7, 9}; + case I64, U64 -> new long[]{7, 8, 0, 7, 9}; + case F32 -> new float[]{7, 8, 0, 7, 9}; + case F64 -> new double[]{7, 8, 0, 7, 9}; + }; + boolean[] validity = {true, true, false, true, true}; + var sut = new DictColumnState(new DType.Primitive(ptype, true)); // When - boolean result = DictColumnState.admitsPtype(ptype); + int ingested = sut.ingestDictChunk(new NullableData(values, validity)); + Object result = sut.reconstructChunk(sut.buildInverseMap(), 0); - // Then - assertThat(result).isEqualTo(expected); + // Then — every row ingested, three values plus the null entry, the chunk rebuilt exactly + assertThat(ingested).isEqualTo(5); + assertThat(java.lang.reflect.Array.getLength(sut.uniques())).isEqualTo(4); + assertThat(result).isInstanceOf(NullableData.class); + assertThat(((NullableData) result).values()).isEqualTo(values); + assertThat(((NullableData) result).validity()).isEqualTo(validity); } // ── isVarBinDictCandidate ────────────────────────────────────────────────────── diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java index 564beb772..ba6bbfb85 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/GlobalDictPrimitiveTest.java @@ -372,12 +372,11 @@ void lowCardinality_i32_usesGlobalDict(@TempDir Path tmp) throws IOException { } @Test - void lowCardinality_i16_notDicted_roundTrips(@TempDir Path tmp) throws IOException { - // Regression: the writer used to admit I16 to the global dict, producing a vortex.dict - // column the reader cannot decode ("unsupported ptype for lazy dict: I16"). I8/I16 are now - // excluded from dict candidacy — matching the Rust compressor, which does not dict narrow - // ints (RustWritesJavaReadsIntegrationTest#jniWriter_javaReader_lowCardinalityI16) — so a - // low-card I16 column encodes via the cascade and round-trips cleanly. + void lowCardinality_i16_roundTrips(@TempDir Path tmp) throws IOException { + // Regression: an I16 dict layout once made a file the reader could not decode + // ("unsupported ptype for lazy dict: I16"). Every primitive is a dict-layout candidate + // again, as in Rust (#651), so a low-cardinality I16 column must read back as I16 + // whichever layout the probe picks. var schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(DTypes.I16), false); short[] data = {1, 2, 3, 1, 2, 3, 1, 2}; Path file = tmp.resolve("i16.vortex"); @@ -386,7 +385,7 @@ void lowCardinality_i16_notDicted_roundTrips(@TempDir Path tmp) throws IOExcepti sut.writeChunk(Map.of(ColumnName.of("v"), data)); } - // When — I16 now encodes via the cascade (a ShortArray view), not a dict + // When short[] result; try (var vf = VortexReader.open(file, ReadRegistry.loadAll()); var iter = vf.scan(io.github.dfa1.vortex.reader.ScanOptions.columns(ColumnName.of("v")))) {