diff --git a/AGENTS.md b/AGENTS.md index d8890c758..b273bc966 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -53,7 +53,7 @@ reader — VortexReader, VortexHttpReader, VortexHandle, ReadRegistry, Chunk, A reader.extension — ExtensionDecoder + Date/Time/Timestamp/Uuid impls reader.layout — Layout, LayoutDecoder, LayoutDecodeContext, LayoutRegistry + built-in *LayoutDecoder impls, ZonedStatsSchema -writer — VortexWriter, WriteRegistry, WriteOptions, ColumnEncoding, ExtensionEncoder +writer — VortexWriter, WriteSummary, WriteRegistry, WriteOptions, ColumnEncoding, ExtensionEncoder VortexWriter orchestrates package-private parts (one concern each): SegmentEncoder — picks and runs encoders; immutable, the only state executor tasks touch SegmentSink — ordered segment writes to the channel, footer, postscript, trailer diff --git a/CHANGELOG.md b/CHANGELOG.md index 54d341c12..ecaddf00b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - JDK Flight Recorder events for file open, HTTP range requests, segment decode and encode, chunk filtering, writer back-pressure and the compressor's scheme choices, which mirror Rust's compressor trace. ([#601](https://github.com/dfa1/vortex-java/issues/601)) +- `VortexWriter.finish()` returns a `WriteSummary` with the file size, row count and each column's compressed size and statistics, so a write needs no read to learn them; `close()` after it does nothing. ([#664](https://github.com/dfa1/vortex-java/issues/664)) ## [0.19.1] — 2026-11-07 diff --git a/docs/explanation.md b/docs/explanation.md index d4000ddf2..e9ccddfcc 100644 --- a/docs/explanation.md +++ b/docs/explanation.md @@ -500,7 +500,7 @@ writer.writeChunk(c -> c.put(column, data[]).put(...)) └─ write buffers to FileChannel, record SegmentSpec (offset + length) └─ record Layout node (encoding ID + rowCount + segment index) -writer.close() +writer.finish() (or close()) → WriteSummary: file size, rows, per-column size + stats └─ write DType blob (FlatBuffer) └─ write Footer blob (FlatBuffer) → SegmentSpec[] + ArraySpec[] └─ write Layout blob (FlatBuffer) → Struct → Zoned(Stats) → Chunked → [Flat …] diff --git a/docs/reference.md b/docs/reference.md index dcac242a0..3ac5020fa 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -136,7 +136,7 @@ The default `HttpClient` is a single JDK client shared across every `VortexHttpR ### `VortexWriter` (`io.github.dfa1.vortex.writer.VortexWriter`) -Writes a Vortex file. Implements `Closeable`. The file is complete and readable as soon as `close()` returns. +Writes a Vortex file. Implements `Closeable`. The file is complete and readable as soon as `finish()` or `close()` returns. | Method | Notes | |----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| @@ -145,7 +145,8 @@ Writes a Vortex file. Implements `Closeable`. The file is complete and readable | `static create(WritableByteChannel, DType.Struct, WriteOptions, WriteRegistry)` | Custom [`WriteRegistry`](#writeregistry-iogithubdfa1vortexwriterwriteregistry) — encoders and extension encoders together | | `writeChunk(Consumer)` | One batch of rows; typed builder validates column names + array types at each `.put`; missing columns throw `IllegalStateException` when the lambda returns. Preferred when columns are known at compile time. | | `writeChunk(Map)` | One batch of rows by map. Validates that every schema column is present and that all columns share the same row count. Use when the column set is built dynamically (Parquet/JDBC importers, generic exporters). | -| `close()` | Finalizes file (footer, postscript, trailer) | +| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, sum, null and NaN counts, folded over the batches, as vortex-jni's `VortexWriter.finish()` reports them). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` | +| `close()` | `finish()` unless it already ran | Batches are not chunks: as Rust's writer does, each primitive, boolean, decimal, string and binary column is repartitioned into chunks of whole 8192-row blocks reaching 1 MB of uncompressed data @@ -199,7 +200,7 @@ A plain final class: start from a factory below and adjust with the `withXxx` me | `withGlobalDict(boolean)` | Toggle the shared cross-chunk dictionary | | `withCompact(boolean)` | Rust's compact preset (`with_compact`): Zstandard for strings and binary and Pco for integers and floats compete in the cascade. Requires `allowedCascading > 0`; `compact()` reads it back | | `withGlobalDictMaxRetainedBytes(long)` | Aggregate heap budget for buffered global-dict candidate columns | -| `withExecutor(Executor)` | Compress segments on `executor` (e.g. the common `ForkJoinPool`); default is the calling thread. Output is byte-identical to a sequential write. The writer owns arrays passed to `writeChunk` until `close()` — do not modify them | +| `withExecutor(Executor)` | Compress segments on `executor` (e.g. the common `ForkJoinPool`); default is the calling thread. Output is byte-identical to a sequential write. The writer owns arrays passed to `writeChunk` until `finish()` or `close()` — do not modify them | | `withEdition(Edition)` | Enable an [edition](#editions) for its family, replacing any edition already enabled for that family | | `withoutEditions()` | Turn the edition guard off, as Rust's `disable_editions()`: every encoding may be emitted, including those in no edition (`fastlanes.delta`, `vortex.patched`). Files may not be readable by other Vortex versions | | `withColumnEncoding(ColumnName, ColumnEncoding)` | Choose one column's encodings from `ColumnEncoding.candidates(EncodingId...)` only — the whole candidate set, children included, with canonical encodings as the fallback (Rust's `with_field_writer` over a restricted scheme set). The edition guard still applies and the column skips the global dict. An unknown column, or an encoding the writer cannot emit, fails `VortexWriter.create` | diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/WriteSummaryInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/WriteSummaryInteropIntegrationTest.java new file mode 100644 index 000000000..2dd403031 --- /dev/null +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/WriteSummaryInteropIntegrationTest.java @@ -0,0 +1,264 @@ +package io.github.dfa1.vortex.integration; + +import dev.vortex.api.Session; +import dev.vortex.api.VortexColumnStatistics; +import dev.vortex.api.VortexWriteSummary; +import dev.vortex.arrow.ArrowAllocation; +import dev.vortex.jni.NativeLoader; +import io.github.dfa1.vortex.core.model.ArrayStats; +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.reader.VortexReader; +import io.github.dfa1.vortex.reader.layout.Layout; +import io.github.dfa1.vortex.writer.VortexWriter; +import io.github.dfa1.vortex.writer.WriteOptions; +import io.github.dfa1.vortex.writer.WriteSummary; +import org.apache.arrow.c.ArrowArray; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.BigIntVector; +import org.apache.arrow.vector.BitVector; +import org.apache.arrow.vector.DecimalVector; +import org.apache.arrow.vector.Float8Vector; +import org.apache.arrow.vector.IntVector; +import org.apache.arrow.vector.UInt8Vector; +import org.apache.arrow.vector.VarBinaryVector; +import org.apache.arrow.vector.VarCharVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.types.FloatingPointPrecision; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.math.BigDecimal; +import java.nio.channels.FileChannel; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.HexFormat; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.OptionalLong; + +import static org.assertj.core.api.Assertions.assertThat; + +/// [VortexWriter#finish()]'s summary against vortex-jni's `VortexWriter.finish()` for the same batches +/// (#664): the row count and, per column, the null and NaN counts and the bounds, which both take +/// from the footer's file statistics. The compressed sizes depend on each writer's encodings, so +/// those are checked against the file instead: Rust's `compressed_column_sizes` sums the segments +/// of each column's layout subtree. +class WriteSummaryInteropIntegrationTest { + + private static final Session SESSION = Session.create(); + private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator(); + private static final int ROWS = 1000; + private static final int BATCHES = 2; + private static final List NAMES = List.of("i64", "i32n", "f64", "u64", "utf8", "bin", "booln", "dec") + .stream().map(ColumnName::of).toList(); + + @BeforeAll + static void loadJni() { + NativeLoader.loadJni(); + } + + @Test + void javaSummaryReportsWhatJniReports(@TempDir Path tmp) throws Exception { + // Given + VortexWriteSummary jni = writeJni(tmp.resolve("jni.vortex")); + + // When + WriteSummary result = writeJava(tmp.resolve("java.vortex")); + + // Then + assertThat(result.rowCount()).isEqualTo(jni.rowCount()); + assertThat(columns(result)).isEqualTo(columns(jni)); + } + + @Test + void sizesAreTheFilesSizes(@TempDir Path tmp) throws Exception { + // Given + Path file = tmp.resolve("java.vortex"); + + // When + WriteSummary result = writeJava(file); + + // Then — the file size includes the trailer, and each column owns its layout subtree's segments + assertThat(result.fileSize()).isEqualTo(Files.size(file)); + try (VortexReader reader = VortexReader.open(file)) { + List expected = new ArrayList<>(); + for (Layout column : reader.layout().children()) { + expected.add(segmentBytes(column, reader)); + } + assertThat(result.columns().values()).extracting(WriteSummary.Column::compressedSize) + .isEqualTo(expected); + } + } + + // ── data ──────────────────────────────────────────────────────────────── + + private static boolean valid(int row) { + return row % 7 != 0; + } + + private static double f64(int row) { + return row % 11 == 0 ? Double.NaN : row * 0.5; + } + + private static String utf8(int row) { + // a value past 64 bytes: its bound is cut and inexact, and both writers report the cut one + return row == 3 ? "a".repeat(80) : "m" + row; + } + + private static byte[] bin(int row) { + return new byte[]{1, (byte) row}; + } + + private static BigDecimal dec(int row) { + return valid(row) ? BigDecimal.valueOf(row * 101L, 2) : null; + } + + // ── writers ───────────────────────────────────────────────────────────── + + private static VortexWriteSummary writeJni(Path file) throws Exception { + Schema arrow = new Schema(List.of( + Field.notNullable("i64", new ArrowType.Int(64, true)), + Field.nullable("i32n", new ArrowType.Int(32, true)), + Field.notNullable("f64", new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE)), + Field.notNullable("u64", new ArrowType.Int(64, false)), + Field.notNullable("utf8", ArrowType.Utf8.INSTANCE), + Field.notNullable("bin", ArrowType.Binary.INSTANCE), + Field.nullable("booln", ArrowType.Bool.INSTANCE), + Field.nullable("dec", new ArrowType.Decimal(10, 2, 128)))); + try (dev.vortex.api.VortexWriter w = dev.vortex.api.VortexWriter.builder(SESSION, + file.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build()) { + for (int batch = 0; batch < BATCHES; batch++) { + try (VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) { + root.allocateNew(); + for (int i = 0; i < ROWS; i++) { + int row = batch * ROWS + i; + ((BigIntVector) root.getVector("i64")).setSafe(i, row * 3L - 100); + IntVector i32n = (IntVector) root.getVector("i32n"); + BitVector booln = (BitVector) root.getVector("booln"); + DecimalVector dec = (DecimalVector) root.getVector("dec"); + if (valid(row)) { + i32n.setSafe(i, row % 1000); + booln.setSafe(i, row % 2); + dec.setSafe(i, dec(row)); + } else { + i32n.setNull(i); + booln.setNull(i); + dec.setNull(i); + } + ((Float8Vector) root.getVector("f64")).setSafe(i, f64(row)); + ((UInt8Vector) root.getVector("u64")).setSafe(i, row * 7L); + ((VarCharVector) root.getVector("utf8")).setSafe(i, utf8(row).getBytes(StandardCharsets.UTF_8)); + ((VarBinaryVector) root.getVector("bin")).setSafe(i, bin(row)); + } + root.setRowCount(ROWS); + try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema schema = ArrowSchema.allocateNew(ALLOCATOR)) { + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, schema); + w.writeBatch(array.memoryAddress(), schema.memoryAddress()); + } + } + } + return w.finish(); + } + } + + private static WriteSummary writeJava(Path file) throws IOException { + DType.Struct schema = new DType.Struct(NAMES, List.of(DTypes.I64, DTypes.I32.asNullable(), DTypes.F64, + DTypes.U64, DTypes.UTF8, DTypes.BINARY, DTypes.BOOL.asNullable(), DType.Decimal.of(10, 2, true)), false); + try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + VortexWriter w = VortexWriter.create(ch, schema, WriteOptions.defaults())) { + for (int batch = 0; batch < BATCHES; batch++) { + long[] i64 = new long[ROWS]; + Integer[] i32n = new Integer[ROWS]; + double[] f64 = new double[ROWS]; + long[] u64 = new long[ROWS]; + String[] utf8 = new String[ROWS]; + byte[][] bin = new byte[ROWS][]; + Boolean[] booln = new Boolean[ROWS]; + BigDecimal[] dec = new BigDecimal[ROWS]; + for (int i = 0; i < ROWS; i++) { + int row = batch * ROWS + i; + i64[i] = row * 3L - 100; + i32n[i] = valid(row) ? row % 1000 : null; + f64[i] = f64(row); + u64[i] = row * 7L; + utf8[i] = utf8(row); + bin[i] = bin(row); + booln[i] = valid(row) ? row % 2 == 1 : null; + dec[i] = dec(row); + } + List values = List.of(i64, i32n, f64, u64, utf8, bin, booln, dec); + Map chunk = new LinkedHashMap<>(); + for (int c = 0; c < NAMES.size(); c++) { + chunk.put(NAMES.get(c), values.get(c)); + } + w.writeChunk(chunk); + } + return w.finish(); + } + } + + // ── comparison ────────────────────────────────────────────────────────── + + private static List columns(WriteSummary summary) { + List out = new ArrayList<>(); + for (Map.Entry e : summary.columns().entrySet()) { + ArrayStats s = e.getValue().stats(); + out.add(e.getKey() + ": nulls=" + s.nullCount() + " nans=" + s.nanCount() + + " lower=" + text(s.min()) + " upper=" + text(s.max())); + } + return out; + } + + private static List columns(VortexWriteSummary summary) { + List out = new ArrayList<>(); + for (VortexColumnStatistics c : summary.columnStatistics()) { + out.add(NAMES.get(c.columnIndex()) + ": nulls=" + count(c.nullValueCount()) + " nans=" + count(c.nanValueCount()) + + " lower=" + text(c.lowerBound()) + " upper=" + text(c.upperBound())); + } + return out; + } + + private static Long count(OptionalLong count) { + return count.isPresent() ? count.getAsLong() : null; + } + + private static String text(Optional value) { + return text(value.orElse(null)); + } + + /// One spelling per value for both writers: vortex-jni boxes integers by width (`Integer`, + /// `Long`, `BigInteger`) where the summary has a `Long`, and its decimals carry their own scale. + private static String text(Object value) { + return switch (value) { + case null -> "-"; + case byte[] bytes -> HexFormat.of().formatHex(bytes); + case BigDecimal decimal -> decimal.stripTrailingZeros().toPlainString(); + default -> value.toString(); + }; + } + + private static long segmentBytes(Layout layout, VortexReader reader) { + long bytes = 0; + for (int segment : layout.segments()) { + bytes += reader.footer().segmentSpecs().get(segment).length(); + } + for (Layout child : layout.children()) { + bytes += segmentBytes(child, reader); + } + return bytes; + } +} diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java b/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java index 27a78c150..a21d8fdeb 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java @@ -1,5 +1,6 @@ package io.github.dfa1.vortex.writer; +import io.github.dfa1.vortex.core.model.ArrayStats; import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.PType; import io.github.dfa1.vortex.core.proto.ProtoScalarValue; @@ -123,12 +124,57 @@ NodeStats stats() { if (summable() && !sumOverflowed) { sumScalar = sumScalar(sum); } - Long nans = null; + return new NodeStats(scalar(min), scalar(maxValue()), null, null, null, nullCount, null, nans(), sumScalar); + } + + /// The folded stats as Java values: a `Long` for an integer (a `U64` as its bits) or a bool sum, + /// a `Float` for an `F16`/`F32` bound and a `Double` for an `F64` one or any float sum, a + /// `Boolean`, a `BigDecimal`, a `String` for UTF-8 and a `byte[]` for binary. + /// + /// @return the same stats as [#stats()]; [ArrayStats#empty()] where that is [NodeStats#NONE] + ArrayStats arrayStats() { + if (!seenBatch) { + return ArrayStats.empty(); + } + Object sumValue = null; + if (summable() && !sumOverflowed) { + sumValue = sum; + } + Object lo = javaValue(min); + Object hi = javaValue(maxValue()); + return new ArrayStats(lo, lo != null && !minTruncated, hi, hi != null && !maxTruncated, sumValue, + null, null, null, nullCount, null, nans()); + } + + private Object maxValue() { + if (maxDropped) { + return null; + } + return max; + } + + private Long nans() { if (!extension && (kind == Kind.F16 || kind == Kind.FLOAT)) { - nans = nanCount; + return nanCount; + } + return null; + } + + private Object javaValue(Object value) { + if (value == null) { + return null; } - Object maxValue = maxDropped ? null : max; - return new NodeStats(scalar(min), scalar(maxValue), null, null, null, nullCount, null, nans, sumScalar); + return switch (kind) { + case F16 -> ((Double) value).floatValue(); + case FLOAT -> { + if (ptype == PType.F32) { + yield ((Double) value).floatValue(); + } + yield value; + } + case UTF8 -> new String((byte[]) value, StandardCharsets.UTF_8); + default -> value; + }; } /// Whether the min is the column's exact minimum: no batch cut its bound. diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java b/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java index 771f15e26..169f95754 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentSink.java @@ -61,6 +61,8 @@ final class SegmentSink { private final SegmentEncoder encoder; private final Executor executor; private final List segs; + // the column each submitted segment belongs to, by segment index + private final List segColumns; private final Map encodingIdx; private final ArrayDeque> pending; private long bytesWritten; @@ -71,6 +73,7 @@ final class SegmentSink { this.encoder = encoder; this.executor = executor; this.segs = new ArrayList<>(); + this.segColumns = new ArrayList<>(); this.encodingIdx = new LinkedHashMap<>(); this.pending = new ArrayDeque<>(); this.bytesWritten = 0; @@ -88,6 +91,7 @@ int write(ColumnName column, DType dtype, Object data, EncodingEncoder encodingO EncodeTrace trace = EncodeTrace.of(column.value(), segIdx); pending.add(CompletableFuture.supplyAsync( () -> encoder.encode(trace, dtype, data, encodingOverride, excludedFromCascade), executor)); + segColumns.add(column); submittedSegments++; drain(MAX_IN_FLIGHT_SEGMENTS); return segIdx; @@ -109,7 +113,8 @@ SegmentStats stats(int segIdx) { /// @param zonedLayout the layout id [LayoutTree]'s zoned nodes stand for /// @param fileStats one accumulator per top-level field, in schema order; a schema with no /// field gets no statistics segment, as Rust writes none - void finish(LayoutId zonedLayout, ByteBuffer dtype, ByteBuffer layout, List fileStats) + /// @return the size of the file in bytes + long finish(LayoutId zonedLayout, ByteBuffer dtype, ByteBuffer layout, List fileStats) throws IOException { drain(0); ByteBuffer footerBuf = buildFooter(zonedLayout); @@ -144,6 +149,19 @@ void finish(LayoutId zonedLayout, ByteBuffer dtype, ByteBuffer layout, List columnSizes() { + Map sizes = new LinkedHashMap<>(); + for (int i = 0; i < segs.size(); i++) { + sizes.merge(segColumns.get(i), segs.get(i).len(), Long::sum); + } + return sizes; } /// Rethrows a failed task's unchecked cause as the sequential path would have thrown it (encoders diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java index c65d8fd91..e144b64bc 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java @@ -72,6 +72,10 @@ public final class VortexWriter implements Closeable { // The batch being written: its columns' piece sizes, measured on the executor ahead of appendChunk. private final Map> pieceSizes; private boolean firstChunkSeen; + private long rowCount; + // set when finish() starts: the file takes no more chunks, and a failed finish is not retried + private boolean finishing; + private WriteSummary summary; private VortexWriter( WritableByteChannel channel, DType.Struct schema, WriteOptions options, List encodings, @@ -104,6 +108,9 @@ private VortexWriter( this.repartitioners = new LinkedHashMap<>(); this.pieceSizes = new HashMap<>(); this.firstChunkSeen = false; + this.rowCount = 0; + this.finishing = false; + this.summary = null; } /// Whether the write may emit Rust's `vortex.zoned` layout, added in `core2026.08.0`: with the @@ -234,7 +241,12 @@ public void writeChunk(Consumer builder) throws IOException { /// @throws IOException if an I/O error occurs writing to the underlying channel /// @throws IllegalArgumentException if a schema column is missing from `columns`, /// or if column arrays disagree on row count + /// @throws IllegalStateException if the writer is already finished public void writeChunk(Map columns) throws IOException { + if (finishing) { + // the trailer is written: more rows would land after it, outside the file + throw new IllegalStateException("writer is finished"); + } // Adapt each column up front so the map entry point accepts the same shapes as the // builder: boxed nullable arrays (Long[], Integer[], …) become NullableData, // raw primitive arrays pass through. Done before the row-count check so length validation @@ -268,6 +280,7 @@ public void writeChunk(Map columns) throws IOException { } } + rowCount += Math.max(expectedLen, 0); pieceSizes.clear(); // Route extension columns first, so the columns' batches can be sized in parallel below. for (int i = 0; i < schema.fieldNames().size(); i++) { @@ -355,8 +368,30 @@ private void flushRepartitioners() throws IOException { } } - @Override - public void close() throws IOException { + /// Completes the file — the last chunks, zone maps, footer and trailer — and returns what it + /// holds, as Rust's `VortexWriter::finish` does. Later calls return the same summary; the + /// writer then takes no more chunks. + /// + /// ```java + /// WriteSummary summary; + /// try (VortexWriter writer = VortexWriter.create(channel, schema, WriteOptions.defaults())) { + /// writer.writeChunk(columns); + /// summary = writer.finish(); + /// } + /// ``` + /// + /// @return the file's size, row count and per-column sizes and statistics + /// @throws IOException if an I/O error occurs writing to the underlying channel + /// @throws IllegalStateException if an earlier call failed: the file is incomplete + public WriteSummary finish() throws IOException { + if (summary != null) { + return summary; + } + if (finishing) { + // writing the footer again would append a second one after the partial first + throw new IllegalStateException("an earlier finish failed; the file is incomplete"); + } + finishing = true; flushRepartitioners(); globalDict.flush(); zoneMaps.flush(colChunks, globalDict.written()); @@ -366,8 +401,24 @@ public void close() throws IOException { if (zonedLayout) { zoned = LayoutId.ZONED; } - sink.finish(zoned, dtypeSerializer.buildDType(schema), + long fileSize = sink.finish(zoned, dtypeSerializer.buildDType(schema), LayoutTree.build(schema, colChunks, globalDict.written(), zoneMaps.written()), fileStats); + Map sizes = sink.columnSizes(); + Map columns = new LinkedHashMap<>(); + for (int i = 0; i < schema.fieldNames().size(); i++) { + ColumnName name = schema.fieldNames().get(i); + columns.put(name, new WriteSummary.Column(sizes.getOrDefault(name, 0L), fileStats.get(i).arrayStats())); + } + summary = new WriteSummary(fileSize, rowCount, columns); + return summary; + } + + /// Finishes the file unless [#finish()] already did. + /// + /// @throws IOException if an I/O error occurs writing to the underlying channel + @Override + public void close() throws IOException { + finish(); } private int writeSegment(ColumnName column, DType dtype, Object data, EncodingEncoder encodingOverride) diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java new file mode 100644 index 000000000..73d70a295 --- /dev/null +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteSummary.java @@ -0,0 +1,36 @@ +package io.github.dfa1.vortex.writer; + +import io.github.dfa1.vortex.core.model.ArrayStats; +import io.github.dfa1.vortex.core.model.ColumnName; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +/// What a finished write produced, returned by [VortexWriter#finish()]: Rust's `WriteSummary`, so a +/// caller learns the file's size and statistics without opening it again. +/// +/// @param fileSize the size of the file in bytes, trailer included +/// @param rowCount the number of rows written +/// @param columns one entry per top-level column, in schema order +public record WriteSummary(long fileSize, long rowCount, Map columns) { + + /// Copies `columns`, keeping its order. + public WriteSummary { + columns = Collections.unmodifiableMap(new LinkedHashMap<>(columns)); + } + + /// One top-level column of the written file. + /// + /// `stats` are the file statistics written into the footer: Rust's min, max, sum, null count + /// and NaN count, each folded over the `writeChunk` batches. A UTF-8 or binary bound is cut to + /// 64 bytes and then inexact (`minExact`/`maxExact` say so); a binary bound is a `byte[]`. A + /// statistic Rust does not keep for the column's type is `null`, as is every one of a column + /// without rows. + /// + /// @param compressedSize the bytes of every segment of the column: data, zone map and + /// dictionary, as Rust's `compressed_column_sizes` counts them + /// @param stats the column's file statistics + public record Column(long compressedSize, ArrayStats stats) { + } +} diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java index d83c4c171..feebc8f93 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java @@ -1,5 +1,6 @@ package io.github.dfa1.vortex.writer; +import io.github.dfa1.vortex.core.model.ArrayStats; import io.github.dfa1.vortex.core.model.ColumnName; import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.DTypes; @@ -9,6 +10,7 @@ import io.github.dfa1.vortex.reader.VortexReader; import io.github.dfa1.vortex.reader.Chunk; import io.github.dfa1.vortex.reader.ScanOptions; +import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -785,4 +787,73 @@ private static byte[] writeMixedColumns(WriteOptions options) throws IOException } return out.toByteArray(); } + + @Nested + class Finish { + + private final java.io.ByteArrayOutputStream out = new java.io.ByteArrayOutputStream(); + + @Test + void summarizesTheFile() throws IOException { + // Given + VortexWriter sut = VortexWriter.create(java.nio.channels.Channels.newChannel(out), SCHEMA, WriteOptions.defaults()); + sut.writeChunk(Map.of(ColumnName.of("id"), new long[]{3, 1, 2}, ColumnName.of("value"), new double[]{0.5, 1.5, 2.5})); + sut.writeChunk(Map.of(ColumnName.of("id"), new long[]{9}, ColumnName.of("value"), new double[]{-1.0})); + + // When + WriteSummary result = sut.finish(); + + // Then — the stats fold both batches + assertThat(result.fileSize()).isEqualTo(out.size()); + assertThat(result.rowCount()).isEqualTo(4); + assertThat(result.columns()).containsOnlyKeys(ColumnName.of("id"), ColumnName.of("value")); + ArrayStats id = result.columns().get(ColumnName.of("id")).stats(); + assertThat(id).isEqualTo(new ArrayStats(1L, true, 9L, true, 15L, null, null, null, 0L, null, null)); + assertThat(result.columns().get(ColumnName.of("value")).compressedSize()).isPositive(); + } + + @Test + void twice_returnsTheSameSummaryAndWritesNothing() throws IOException { + // Given — close() after finish() used to write a second footer after the first trailer + VortexWriter sut = VortexWriter.create(java.nio.channels.Channels.newChannel(out), SCHEMA, WriteOptions.defaults()); + sut.writeChunk(Map.of(ColumnName.of("id"), new long[]{1}, ColumnName.of("value"), new double[]{1.0})); + WriteSummary first = sut.finish(); + int size = out.size(); + + // When + WriteSummary result = sut.finish(); + sut.close(); + + // Then + assertThat(result).isSameAs(first); + assertThat(out.size()).isEqualTo(size); + } + + @Test + void writeChunkAfterFinish_throws() throws IOException { + // Given — rows written after the trailer would sit outside the file + VortexWriter sut = VortexWriter.create(java.nio.channels.Channels.newChannel(out), SCHEMA, WriteOptions.defaults()); + sut.finish(); + Map chunk = Map.of(ColumnName.of("id"), new long[]{1}, ColumnName.of("value"), new double[]{1.0}); + + // When / Then + assertThatThrownBy(() -> sut.writeChunk(chunk)) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("finished"); + } + + @Test + void noRows_hasEmptyStats() throws IOException { + // Given — Rust keeps no statistic of a column it saw no batch of + VortexWriter sut = VortexWriter.create(java.nio.channels.Channels.newChannel(out), SCHEMA, WriteOptions.defaults()); + + // When + WriteSummary result = sut.finish(); + + // Then + assertThat(result.rowCount()).isZero(); + assertThat(result.columns().values()).extracting(WriteSummary.Column::stats) + .containsOnly(ArrayStats.empty()); + } + } }