Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The child nodes of a flat segment carry the array stats Rust keeps: a sequence its sorted flags, a nullable integer's validity bitmap its true count, and patch values `is_constant = false`. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- The zone table of a column holds Rust's shape: no validity or mask on a column without nulls, and a constant bool stays a `vortex.bool` until the constant is smaller. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- Text columns stay `vortex.varbinview`, Rust's canonical form, unless a scheme is smaller, and a `vortex.varbin` is only chosen for binary. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- A numeric column keeps its dict layout past 2,048 distinct values, admitted by Rust's probe on its first 8,192 rows and starting a new dictionary at 65,535 entries as Rust does. ([#649](https://github.com/dfa1/vortex-java/issues/649))
- A text column keeps its dict layout past 32,768 distinct values, starting a new dictionary at 65,535 entries or 1 MiB as Rust does, and is admitted by its first 8,192 rows. ([#650](https://github.com/dfa1/vortex-java/issues/650))
- The compact preset writes text columns as zstd frames of 8192 values, as Rust does, so a row-range scan decompresses only the frames it overlaps. ([#636](https://github.com/dfa1/vortex-java/issues/636))
- A nullable decimal column keeps its validity in its own encoding, or has none when every row is valid, instead of a `vortex.masked` wrapper, and a column of equal decimals is one constant, as Rust writes them: decimal zone-stats tables are smaller. ([#646](https://github.com/dfa1/vortex-java/issues/646))
Expand Down
2 changes: 1 addition & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.varbinview` | `VarBinViewEncodingDecoder` | `VarBinViewEncodingEncoder` | ✅ | ✅ | Utf8, Binary |
| `vortex.alp` | `AlpEncodingDecoder` | `AlpEncodingEncoder` | ✅ | ✅ | F64, F32 |
| `vortex.alprd` | `AlpRdEncodingDecoder` | `AlpRdEncodingEncoder` | ✅ | ✅ | F64, F32 |
| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and under half distinct values in its first batch, at most 2,048 of them for numbers (Rust has no such cap and skips only above half); a text column has no cap either: it needs under half of its first 8,192 rows distinct, and starts a new dictionary whenever the open one reaches 65,535 entries or 1 MiB, as Rust's dict layout does. A non-null primitive column that passes then gets the layout only when the cascade, with the integer dictionary back in, picks `vortex.dict` for its first rows: Rust's `DictStrategy` probe. Rust probes exactly the first 8192 rows across batches; Java probes the first batch's first 8192, so a first batch under 8192 rows can decide differently. Nullable and string columns skip the probe and decide by the distinct-count check alone, which admits some nullable integer columns Rust bit-packs. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust |
| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and Rust's probe: the cascade, run on the column's first 8,192 rows, must pick `vortex.dict` (Rust's `DictStrategy`). Text columns skip the probe, as our cascade picks OnPair where Rust's string dictionary wins, and need under half of those rows distinct. Neither has a cardinality cap: whenever the open dictionary reaches 65,535 entries or 1 MiB the writer closes it and starts the next, and a column of several dictionaries is a chunked layout of dict layouts, as in Rust. With `cascading(0)` there is no cascade to probe, so the distinct count of those rows (under half, and more than one) decides. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust |
| `vortex.sparse` | `SparseEncodingDecoder` | `SparseEncodingEncoder` | ✅ | ✅ | Primitive, Bool, Utf8/Binary (decode); Primitive (encode) |
| `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) |
| `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
Expand All @@ -38,11 +39,12 @@

import static org.assertj.core.api.Assertions.assertThat;

/// Which text columns get Rust's dict layout, and how many dictionaries it makes (#650). Rust's
/// `DictStrategy` admits a column by its first 8192 rows (not more than half distinct) and starts a
/// new dictionary whenever the open one reaches 65,535 entries or 1 MiB, so a column of any total
/// cardinality can keep its dict layout.
class DictLayoutStringInteropIntegrationTest {
/// Which text and numeric columns get Rust's dict layout, and how many dictionaries it makes (#650,
/// #649). Rust's `DictStrategy` admits a column by its first 8192 rows and starts a new dictionary
/// whenever the open one reaches 65,535 entries or 1 MiB, so a column of any total cardinality can
/// keep its dict layout.
@Tag("slow")
class DictLayoutInteropIntegrationTest {

private static final Session SESSION = Session.create();
private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator();
Expand Down Expand Up @@ -100,6 +102,96 @@ void startsANewDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws Exceptio
assertThat(readAll(java)).containsExactly(values);
}

@ParameterizedTest(name = "pool of {0} values")
@ValueSource(ints = {100, 3_000, 20_000, 150_000})
void admitsTheNumericColumnsRustAdmits(int poolSize, @TempDir Path tmp) throws Exception {
// Given — 3 chunks of full-width i64 drawn at random from a pool (hashes, like ClickBench's URLHash)
Random r = new Random(11);
long[] pool = new long[poolSize];
for (int i = 0; i < pool.length; i++) {
pool[i] = r.nextLong();
}
long[] values = new long[CHUNK * 3];
for (int i = 0; i < values.length; i++) {
values[i] = pool[r.nextInt(pool.length)];
}

// When
Path java = writeJava(tmp, values);
Path jni = writeJni(tmp, values);

// Then
assertThat(dictLayouts(EncodingTrees.of(java))).isEqualTo(dictLayouts(EncodingTrees.of(jni)));
}

@Test
void startsANewNumericDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws Exception {
// Given — 8 chunks: the first 8192 rows hold 2000 distinct values, every row after them is new
Random r = new Random(5);
long[] values = new long[CHUNK * 8];
for (int i = 0; i < values.length; i++) {
values[i] = i < 8_192 ? r.nextLong() % 2_000 * 1_000_003L : r.nextLong();
}

// When
Path java = writeJava(tmp, values);
Path jni = writeJni(tmp, values);

// Then — whatever Rust decides for this column, so do we, dictionary for dictionary
assertThat(dictLayouts(EncodingTrees.of(java))).isEqualTo(dictLayouts(EncodingTrees.of(jni)));
assertThat(readAllLongs(java)).containsExactly(values);
}

private static Path writeJava(Path tmp, long[] values) throws IOException {
Path file = tmp.resolve("java.vortex");
DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.I64), false);
try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
var w = io.github.dfa1.vortex.writer.VortexWriter.create(ch, schema, WriteOptions.defaults())) {
for (int from = 0; from < values.length; from += CHUNK) {
w.writeChunk(Map.of(ColumnName.of("url"), java.util.Arrays.copyOfRange(values, from, from + CHUNK)));
}
}
return file;
}

private static Path writeJni(Path tmp, long[] values) throws Exception {
Path file = tmp.resolve("jni.vortex");
Schema arrow = new Schema(List.of(Field.notNullable("url", new ArrowType.Int(64, true))));
try (VortexWriter w = VortexWriter.builder(SESSION, file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) {
for (int from = 0; from < values.length; from += CHUNK) {
try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) {
org.apache.arrow.vector.BigIntVector v = (org.apache.arrow.vector.BigIntVector) root.getVector("url");
root.allocateNew();
for (int i = 0; i < CHUNK; i++) {
v.setSafe(i, values[from + i]);
}
root.setRowCount(CHUNK);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema as = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, as);
w.writeBatch(array.memoryAddress(), as.memoryAddress());
}
}
}
}
return file;
}

private static long[] readAllLongs(Path file) throws IOException {
List<Long> out = new ArrayList<>();
try (var reader = VortexReader.open(file, ReadRegistry.loadAll());
var iter = reader.scan(ScanOptions.columns(ColumnName.of("url")))) {
while (iter.hasNext()) {
try (var chunk = iter.next()) {
var longs = (io.github.dfa1.vortex.reader.array.LongArray) chunk.column(ColumnName.of("url"));
for (long i = 0; i < longs.length(); i++) {
out.add(longs.getLong(i));
}
}
}
}
return out.stream().mapToLong(Long::longValue).toArray();
}

private static Path writeJava(Path tmp, String[] values) throws IOException {
Path file = tmp.resolve("java.vortex");
DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.UTF8), false);
Expand Down
Loading
Loading