Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The child nodes of a flat segment carry the array stats Rust keeps: a sequence its sorted flags, a nullable integer's validity bitmap its true count, and patch values `is_constant = false`. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- The zone table of a column holds Rust's shape: no validity or mask on a column without nulls, and a constant bool stays a `vortex.bool` until the constant is smaller. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- Text columns stay `vortex.varbinview`, Rust's canonical form, unless a scheme is smaller, and a `vortex.varbin` is only chosen for binary. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- A text column keeps its dict layout past 32,768 distinct values, starting a new dictionary at 65,535 entries or 1 MiB as Rust does, and is admitted by its first 8,192 rows. ([#650](https://github.com/dfa1/vortex-java/issues/650))
- The compact preset writes text columns as zstd frames of 8192 values, as Rust does, so a row-range scan decompresses only the frames it overlaps. ([#636](https://github.com/dfa1/vortex-java/issues/636))
- A nullable decimal column keeps its validity in its own encoding, or has none when every row is valid, instead of a `vortex.masked` wrapper, and a column of equal decimals is one constant, as Rust writes them: decimal zone-stats tables are smaller. ([#646](https://github.com/dfa1/vortex-java/issues/646))
- `VortexAggregates` summarizes a `U64` column exactly: `MIN`, `MAX` and `SUM` compare and add unsigned instead of failing past 2^63. ([#665](https://github.com/dfa1/vortex-java/issues/665))
Expand Down
2 changes: 1 addition & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.varbinview` | `VarBinViewEncodingDecoder` | `VarBinViewEncodingEncoder` | ✅ | ✅ | Utf8, Binary |
| `vortex.alp` | `AlpEncodingDecoder` | `AlpEncodingEncoder` | ✅ | ✅ | F64, F32 |
| `vortex.alprd` | `AlpRdEncodingDecoder` | `AlpRdEncodingEncoder` | ✅ | ✅ | F64, F32 |
| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs under half distinct values in its first batch, at most 2,048 of them (32,768 for strings), and a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`); Rust has no such cap and skips only above half. A non-null primitive column that passes then gets the layout only when the cascade, with the integer dictionary back in, picks `vortex.dict` for its first rows: Rust's `DictStrategy` probe. Rust probes exactly the first 8192 rows across batches; Java probes the first batch's first 8192, so a first batch under 8192 rows can decide differently. Nullable and string columns skip the probe and decide by the cardinality check alone, which admits some nullable integer columns Rust bit-packs. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust |
| `vortex.dict` | `DictEncodingDecoder` | `DictEncodingEncoder` | ✅ | ✅ | Primitive, Utf8/Binary. Writes `DictMetadata` + children `[codes, values]`, both cascaded (a float pool goes through ALP). With `globalDict` on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes `IntDictScheme` there and dictionary-encodes suitable columns once, as a layout. A column first needs a type the dict layout reads (no `I8`/`U8`/`I16`/`U16`/`F16`/`F32`) and under half distinct values in its first batch, at most 2,048 of them for numbers (Rust has no such cap and skips only above half); a text column has no cap either: it needs under half of its first 8,192 rows distinct, and starts a new dictionary whenever the open one reaches 65,535 entries or 1 MiB, as Rust's dict layout does. A non-null primitive column that passes then gets the layout only when the cascade, with the integer dictionary back in, picks `vortex.dict` for its first rows: Rust's `DictStrategy` probe. Rust probes exactly the first 8192 rows across batches; Java probes the first batch's first 8192, so a first batch under 8192 rows can decide differently. Nullable and string columns skip the probe and decide by the distinct-count check alone, which admits some nullable integer columns Rust bit-packs. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, `[values, codes]`) so older Java-written files stay readable; that shape was never readable by Rust |
| `vortex.sparse` | `SparseEncodingDecoder` | `SparseEncodingEncoder` | ✅ | ✅ | Primitive, Bool, Utf8/Binary (decode); Primitive (encode) |
| `vortex.sequence` | `SequenceEncodingDecoder` | `SequenceEncodingEncoder` | ✅ | ✅ | Primitive (encode: integer only, as Rust; decode also float) |
| `vortex.struct` | `StructEncodingDecoder` | `StructEncodingEncoder` | ✅ | ✅ | Struct |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
package io.github.dfa1.vortex.integration;

import dev.vortex.api.Session;
import dev.vortex.api.VortexWriter;
import dev.vortex.arrow.ArrowAllocation;
import dev.vortex.jni.NativeLoader;
import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.core.model.DTypes;
import io.github.dfa1.vortex.reader.ReadRegistry;
import io.github.dfa1.vortex.reader.ScanOptions;
import io.github.dfa1.vortex.reader.VortexReader;
import io.github.dfa1.vortex.reader.array.VarBinArray;
import io.github.dfa1.vortex.writer.WriteOptions;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.VarCharVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Random;

import static org.assertj.core.api.Assertions.assertThat;

/// Which text columns get Rust's dict layout, and how many dictionaries it makes (#650). Rust's
/// `DictStrategy` admits a column by its first 8192 rows (not more than half distinct) and starts a
/// new dictionary whenever the open one reaches 65,535 entries or 1 MiB, so a column of any total
/// cardinality can keep its dict layout.
class DictLayoutStringInteropIntegrationTest {

private static final Session SESSION = Session.create();
private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator();
private static final int CHUNK = 65_536;

static {
NativeLoader.loadJni();
}

@ParameterizedTest(name = "pool of {0} values")
@ValueSource(ints = {100, 3_000, 20_000, 150_000})
void admitsTheColumnsRustAdmits(int poolSize, @TempDir Path tmp) throws Exception {
// Given — 3 chunks drawing at random from a pool: the pool decides how distinct the first 8192 rows are
Random r = new Random(11);
String[] pool = new String[poolSize];
for (int i = 0; i < pool.length; i++) {
pool[i] = "https://example.org/path/" + Long.toString(r.nextLong() & 0xffffffffffL, 36);
}
String[] values = new String[CHUNK * 3];
for (int i = 0; i < values.length; i++) {
values[i] = pool[r.nextInt(pool.length)];
}

// When
Path java = writeJava(tmp, values);
Path jni = writeJni(tmp, values);

// Then
assertThat(dictLayouts(EncodingTrees.of(java))).isEqualTo(dictLayouts(EncodingTrees.of(jni)));
}

@Test
void startsANewDictionaryWhenTheOpenOneIsFull(@TempDir Path tmp) throws Exception {
// Given — 8 chunks, each drawing 3000 values from a window that slides 2500 values on: well under
// half distinct in the first 8192 rows, but 23000 values of 96 bytes overall, past 1 MiB
Random r = new Random(5);
int chunks = 8;
String[] pool = new String[2500 * chunks + 3000];
for (int i = 0; i < pool.length; i++) {
pool[i] = "https://example.org/some/long/path/segment/" + Long.toString(r.nextLong() & 0xffffffffffL, 36)
+ "/index.html?utm=" + i;
}
String[] values = new String[CHUNK * chunks];
for (int i = 0; i < values.length; i++) {
values[i] = pool[(i / CHUNK) * 2500 + r.nextInt(3000)];
}

// When
Path java = writeJava(tmp, values);
Path jni = writeJni(tmp, values);

// Then — Rust fills one dictionary and starts the next, and so do we
assertThat(dictLayouts(EncodingTrees.of(jni))).isEqualTo(2);
assertThat(dictLayouts(EncodingTrees.of(java))).isEqualTo(2);
assertThat(readAll(java)).containsExactly(values);
}

private static Path writeJava(Path tmp, String[] values) throws IOException {
Path file = tmp.resolve("java.vortex");
DType.Struct schema = new DType.Struct(List.of(ColumnName.of("url")), List.of(DTypes.UTF8), false);
try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
var w = io.github.dfa1.vortex.writer.VortexWriter.create(ch, schema, WriteOptions.defaults())) {
for (int from = 0; from < values.length; from += CHUNK) {
String[] chunk = new String[CHUNK];
System.arraycopy(values, from, chunk, 0, CHUNK);
w.writeChunk(Map.of(ColumnName.of("url"), chunk));
}
}
return file;
}

private static Path writeJni(Path tmp, String[] values) throws Exception {
Path file = tmp.resolve("jni.vortex");
Schema arrow = new Schema(List.of(Field.notNullable("url", ArrowType.Utf8.INSTANCE)));
try (VortexWriter w = VortexWriter.builder(SESSION, file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) {
for (int from = 0; from < values.length; from += CHUNK) {
try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) {
VarCharVector v = (VarCharVector) root.getVector("url");
root.allocateNew();
for (int i = 0; i < CHUNK; i++) {
v.setSafe(i, values[from + i].getBytes(StandardCharsets.UTF_8));
}
root.setRowCount(CHUNK);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema as = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, as);
w.writeBatch(array.memoryAddress(), as.memoryAddress());
}
}
}
}
return file;
}

/// How many dict layouts a tree holds: a `vortex.dict` whose child is a layout, not an array encoding.
private static long dictLayouts(List<String> tree) {
long count = 0;
for (int i = 0; i + 1 < tree.size(); i++) {
if (tree.get(i).strip().equals("vortex.dict") && tree.get(i + 1).strip().equals("vortex.flat")) {
count++;
}
}
return count;
}

private static String[] readAll(Path file) throws IOException {
List<String> out = new ArrayList<>();
try (var reader = VortexReader.open(file, ReadRegistry.loadAll());
var iter = reader.scan(ScanOptions.columns(ColumnName.of("url")))) {
while (iter.hasNext()) {
try (var chunk = iter.next()) {
VarBinArray strings = (VarBinArray) chunk.column(ColumnName.of("url"));
for (long i = 0; i < strings.length(); i++) {
out.add(strings.getString(i));
}
}
}
}
return out.toArray(String[]::new);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -94,12 +94,12 @@ void globalDictUtf8_valuesPool_isFsstCompressed_andRoundTrips(@TempDir Path tmp)

@Test
void utf8_cardinalityAbovePriorCap_admittedToGlobalDict(@TempDir Path tmp) throws IOException {
// Given — 8 000 distinct values: above the old 2048 numeric-era cap (so previously demoted
// to per-chunk), below the 32768 Utf8 cap, with >50% dedup so the ratio gate admits it.
// Given — 3 000 distinct values: above the old 2048 numeric-era cap (so previously demoted
// to per-chunk), with under half of the first 8192 rows distinct, which is Rust's gate.
Path file = tmp.resolve("high_card.vortex");
int rowsPerChunk = 20_000;
int chunkCount = 4;
int distinct = 8_000;
int distinct = 3_000;

List<String> expected = new ArrayList<>();
try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
Expand All @@ -113,10 +113,10 @@ void utf8_cardinalityAbovePriorCap_admittedToGlobalDict(@TempDir Path tmp) throw
}

// Then — the column is stored as one global dictionary (a DICT layout node), not a per-chunk
// fallback. Before the type-aware cap a 8 000-distinct column produced no DICT layout.
// fallback.
try (var vf = VortexReader.open(file, ReadRegistry.loadAll())) {
assertThat(hasDictLayout(InspectorTree.build(vf).root()))
.as("8 000-distinct Utf8 column admitted to the global dictionary")
.as("3 000-distinct Utf8 column admitted to the global dictionary")
.isTrue();
}

Expand Down
Loading
Loading