Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- FSST symbol tables are trained as Rust's `fsst-rs` does (its sample, generations, candidate gains and cut), so a text column without a null gets Rust's FSST encoding up to ties of equal gain, its offsets a `vortex.sequence` where Rust's cascade finds one. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- `VortexReader.columnStats()` reports the min and max of a decimal column, and a decimal filter compares by value instead of by the long it truncates to. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650))
- 8-bit, 16-bit and `F32` columns get Rust's dict layout when its probe picks one, so a nullable low-cardinality `SMALLINT` is written up to 30% smaller. ([#651](https://github.com/dfa1/vortex-java/issues/651))
- A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696))
- **Breaking:** a `Double`, `Float` or `BigDecimal` filter operand on an integer column fails with `VortexException`, as vortex-jni does, instead of being truncated so that `x >= 1.5` matched `x = 1`. ([#701](https://github.com/dfa1/vortex-java/issues/701))
- A `vortex.variant` column whose shredded dtype has a control character in a field or variant name fails with `VortexException` instead of `IllegalArgumentException`. ([#694](https://github.com/dfa1/vortex-java/issues/694))
Expand Down
2 changes: 1 addition & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.varbinview` | `VarBinViewEncodingDecoder` | `VarBinViewEncodingEncoder` | ✅ | ✅ | Utf8, Binary |
| `vortex.alp` | `AlpEncodingDecoder` | `AlpEncodingEncoder` | ✅ | ✅ | F64, F32 |
| `vortex.alprd` | `AlpRdEncodingDecoder` | `AlpRdEncodingEncoder` | ✅ | ✅ | F64, F32 |
| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and Rust's probe: the cascade, run on the column's first 8,192 rows, must pick `vortex.dict` (Rust's `DictStrategy`). Text columns skip the probe, as our cascade picks OnPair where Rust's string dictionary wins, and need under half of those rows distinct. Neither has a cardinality cap: whenever the open dictionary reaches 65,535 entries or 1 MiB the writer closes it and starts the next, and a column of several dictionaries is a chunked layout of dict layouts, as in Rust. With `cascading(0)` there is no cascade to probe, so the distinct count of those rows (under half, and more than one) decides. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust |
| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. Any primitive column can qualify, as in Rust's `dict_layout_supported`, by Rust's probe: the cascade, run on the column's first 8,192 rows, must pick `vortex.dict` (Rust's `DictStrategy`). Text columns skip the probe, as our cascade picks OnPair where Rust's string dictionary wins, and need under half of those rows distinct. Neither has a cardinality cap: whenever the open dictionary reaches 65,535 entries or 1 MiB the writer closes it and starts the next, and a column of several dictionaries is a chunked layout of dict layouts, as in Rust. With `cascading(0)` there is no cascade to probe, so the distinct count of those rows (under half, and more than one) decides. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust |
| `vortex.sparse` | `SparseEncodingDecoder` | `SparseEncodingEncoder` | ✅ | ✅ | Primitive, Bool, Utf8/Binary (decode); Primitive (encode) |
| `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) |
| `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,30 +1,42 @@
package io.github.dfa1.vortex.integration;

import dev.vortex.api.DataSource;
import dev.vortex.api.Expression;
import dev.vortex.api.Partition;
import dev.vortex.api.Scan;
import dev.vortex.api.Session;
import dev.vortex.api.VortexWriter;
import dev.vortex.arrow.ArrowAllocation;
import dev.vortex.jni.NativeLoader;
import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.core.model.DTypes;
import io.github.dfa1.vortex.core.model.PType;
import io.github.dfa1.vortex.reader.ReadRegistry;
import io.github.dfa1.vortex.reader.ScanOptions;
import io.github.dfa1.vortex.reader.VortexReader;
import io.github.dfa1.vortex.reader.array.VarBinArray;
import io.github.dfa1.vortex.writer.WriteOptions;
import io.github.dfa1.vortex.writer.encode.NullableData;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.FieldVector;
import org.apache.arrow.vector.SmallIntVector;
import org.apache.arrow.vector.TinyIntVector;
import org.apache.arrow.vector.UInt2Vector;
import org.apache.arrow.vector.VarCharVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
import org.junit.jupiter.params.provider.ValueSource;

import java.io.IOException;
Expand Down Expand Up @@ -142,6 +154,131 @@ void startsANewNumericDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws E
assertThat(readAllLongs(java)).containsExactly(values);
}

@ParameterizedTest(name = "{0}")
@EnumSource(value = PType.class, names = {"I8", "I16", "U16"})
void admitsANullableNarrowColumnAsRustDoes(PType ptype, @TempDir Path tmp) throws Exception {
// Given — the profile of Public BI Generico's `Disco` first 8192 rows (#651): a nullable
// SMALLINT, 40% valid, 62 distinct values in 5..132 with a few far more common than the
// rest, value runs of about 2 rows between null runs of about 12. Rust's dict layout admits
// every primitive and writes it as one values pool plus run-end codes; Java filtered narrow
// widths out before the probe and wrote a 1.44x larger bit-packed column with a validity
// per chunk
Random r = new Random(651);
int[] pool = new int[62];
for (int p = 0; p < pool.length; p++) {
pool[p] = 5 + p * 2;
}
int n = CHUNK * 3;
int[] values = new int[n];
boolean[] valid = new boolean[n];
int i = 0;
while (i < n) {
boolean isValid = r.nextInt(100) >= 22;
int run = isValid ? 1 + r.nextInt(3) : 1 + r.nextInt(23);
int value = pool[Math.min(r.nextInt(pool.length), r.nextInt(pool.length))];
for (int j = 0; j < run && i < n; j++, i++) {
values[i] = value;
valid[i] = isValid;
}
}

// When
Path java = writeJavaNullable(tmp, ptype, values, valid);
Path jni = writeJniNullable(tmp, ptype, values, valid);

// Then — the same layouts and encodings, node for node, and vortex-jni reads Java's file
// back to the same values as its own
assertThat(EncodingTrees.of(java)).isEqualTo(EncodingTrees.of(jni));
assertThat(dictLayouts(EncodingTrees.of(java))).isPositive();
assertThat(readWithJni(java)).isEqualTo(readWithJni(jni));
}

private static Path writeJavaNullable(Path tmp, PType ptype, int[] values, boolean[] valid) throws IOException {
Path file = tmp.resolve("java.vortex");
ColumnName col = ColumnName.of("v");
DType.Struct schema = DType.structBuilder().field(col, new DType.Primitive(ptype, true)).build();
try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
var w = io.github.dfa1.vortex.writer.VortexWriter.create(ch, schema, WriteOptions.defaults())) {
for (int from = 0; from < values.length; from += CHUNK) {
Object chunk = switch (ptype) {
case I8 -> {
byte[] a = new byte[CHUNK];
for (int i = 0; i < CHUNK; i++) {
a[i] = (byte) values[from + i];
}
yield a;
}
default -> {
short[] a = new short[CHUNK];
for (int i = 0; i < CHUNK; i++) {
a[i] = (short) values[from + i];
}
yield a;
}
};
w.writeChunk(Map.of(col, new NullableData(chunk, java.util.Arrays.copyOfRange(valid, from, from + CHUNK))));
}
}
return file;
}

private static Path writeJniNullable(Path tmp, PType ptype, int[] values, boolean[] valid) throws Exception {
Path file = tmp.resolve("jni.vortex");
ArrowType type = switch (ptype) {
case I8 -> new ArrowType.Int(8, true);
case U16 -> new ArrowType.Int(16, false);
default -> new ArrowType.Int(16, true);
};
Schema arrow = new Schema(List.of(Field.nullable("v", type)));
try (VortexWriter w = VortexWriter.builder(SESSION, file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) {
for (int from = 0; from < values.length; from += CHUNK) {
try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) {
FieldVector v = root.getVector("v");
root.allocateNew();
for (int i = 0; i < CHUNK; i++) {
int at = from + i;
if (!valid[at]) {
v.setNull(i);
continue;
}
switch (v) {
case TinyIntVector t -> t.setSafe(i, values[at]);
case UInt2Vector u -> u.setSafe(i, values[at]);
case SmallIntVector s -> s.setSafe(i, values[at]);
default -> throw new IllegalStateException(v.getClass().getName());
}
}
root.setRowCount(CHUNK);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema as = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, as);
w.writeBatch(array.memoryAddress(), as.memoryAddress());
}
}
}
}
return file;
}

/// Every row of column `v` as vortex-jni reads it, `null` for a null row.
private static List<Object> readWithJni(Path file) throws Exception {
List<Object> out = new ArrayList<>();
DataSource ds = DataSource.open(SESSION, file.toAbsolutePath().toUri().toString());
Scan scan = ds.scan(dev.vortex.api.ScanOptions.builder()
.projection(Expression.select(new String[]{"v"}, Expression.root())).build());
while (scan.hasNext()) {
Partition partition = scan.next();
try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) {
while (reader.loadNextBatch()) {
FieldVector vec = reader.getVectorSchemaRoot().getVector("v");
for (int i = 0; i < vec.getValueCount(); i++) {
out.add(vec.getObject(i));
}
}
}
}
return out;
}

private static Path writeJava(Path tmp, long[] values) throws IOException {
Path file = tmp.resolve("java.vortex");
DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.I64), false);
Expand Down
Loading
Loading