Repository navigation
Description
VortexWriter is 1712 lines with ~40 mutable fields and no grouping: encoder selection, the async segment pipeline, file I/O, the global dictionary, zone maps, repartitioning and layout-tree building all share one class and one namespace. The only thing keeping the executor thread-safe is a comment ("touches only immutable writer state").
Where the state actually clusters
| Cluster | Fields | Methods | ~Lines |
|---|---|---|---|
| Encoder selection (immutable, runs on the executor) | encodings, defaultRegistry, cascadeCodecs/cascadeRegistry, probeCodecs/probeRegistry, editionAllowed/editionExcluded, columnEncoders |
buildCascadeCodecs, editionAllowed/Excluded/HasZonedLayout, columnEncoder, zoneTableEncoder, resolveOverride, findEncoder, nullableCapableEncoder, encodeSegment, probeCompressesToDict |
~400 |
| Segment sink (ordered I/O + footer) | channel, bytesWritten, segs, encodingIdx, pending, submittedSegments |
writeSegment×3, drainSegments, discardPending, writeEncoded, registerEncodingIds, write×2, writePadding, bufferPaddings, buildArrayFlatBuffer, buildStats, buildArrayNodeFlatBuffer, buildFooter, buildPostscript, trailer |
~350 |
| Global dictionary | dictCandidates, dictStates, dictColRefs, dictRetainedBytes, dictRetainedTotal, dictRetainedBudget, firstChunkSeen |
candidate detection (inline in writeChunk), evictLargest…, demoteDictColumn, flushDictColumns, writeGlobalDict*, dictPool, writeGlobalDictCodes, buildDictColLayout, buildDictLayoutMetaBytes |
~300 |
| Zone maps | zoneMaps, zoneAccumulators, zoneTails, zonedLayout |
flushZoneMaps, joinZoneTails, emitZoneMap, wrapZoneMap |
~200 |
| Columns (what stays) | schema, options, colChunks, repartitioners, pieceSizes |
create×3, ctor validation, writeChunk, appendChunk, writeDataChunk, flushRepartitioners, buildLayout, close |
~450 |
Redundant state found while mapping:
dictCandidatesis alwaysdictStates.keySet()(added and removed together).dictRetainedBytes+dictRetainedTotalcacheDictColumnState#retainedBytes(), which only changes when that column ingests: the total is a sum over the candidate columns' dictionaries (done in c06eb2ee).drainSegmentsandjoinZoneTailsduplicate theCompletionExceptionunwrap.
Plan
Pure refactor: no public API change, byte-identical output. New types are package-private final classes in io.github.dfa1.vortex.writer, the same way DictColumnState, ZoneAccumulator and Repartitioner already are. One commit per step, each green on ./mvnw verify.
- Cleanup, no moves. Delete the redundant state above. Add one helper for the completion unwrap. Use imports instead of the inline FQNs (
io.github.dfa1.vortex.core.fbs.FbsArray*,…encode.NullableData,java.util.HashSet). Fix the duplicatedcreatejavadoc, the two docs merged oneditionHasZonedLayout, and the orphan comments ("Indices into layout_specs", "Base cascade codec list"). MovecountNullsnext toNullableData. - Extract
SegmentEncoder. This is the immutable encoder-selection cluster, built once in the constructor; the executor lambdas capture it instead ofthis, so thread safety comes from the types rather than a comment. Watch the ordering:columnEncoder's anonymous class currently readseditionExcludedlazily, and that field is assigned aftercolumnEncodersis built. Compute the exclusions first and pass them in explicitly. - Extract
SegmentSink. It owns the channel, the in-flight window, segment and encoding-id bookkeeping, buffer and stats FlatBuffers, the footer, postscript and trailer. API:int submit(Supplier<EncodedSegment>),drain(),stats(segIdx)(for the legacy zone map), andfinish(dtypeBuf, layoutBuf). The edition backstop inregisterEncodingIdstakes the allowed set. - Extract
GlobalDictWriter. It covers detection on the first batch, ingest, eviction and demotion, the flush at close, and its own layout subtree (buildDictColLayout, metadata bytes). Demotion replays chunks into the column path through a smallChunkSinkcallback (appendChunk), not a back-reference to the writer.writeChunkasks itisCandidate(col). - Extract
ZoneMapWriter. It feeds the zoned accumulators per batch, flushes both the zoned and the legacyvortex.statsmaps, and doeswrapZoneMap. The legacy path reads chunk stats fromSegmentSink#statsand dict chunk stats fromGlobalDictWriter.
Target: VortexWriter at about 450 lines. It keeps the public factories, schema validation, writeChunk (adapt, row-count check, extension routing, piece sizing, dispatch), column chunking and repartition, buildLayout, and the close() orchestration. arrayLength stays a package-private static there, since every collaborator uses it.
Out of scope: any behavior change, the MAX_IN_FLIGHT_SEGMENTS knob, and the concurrent-scan work.
Risks
close()order is load-bearing: repartitioner flush, then dict flush, then zone maps, then drain, then footer. Global-dict demotion duringwriteChunkinterleaves segment indexes. Any reordering changes the segment layout of the file.- Hidden ordering in the constructor: see step 2.
- Perf: the hot loops are in the encoders, not here, but the executor closures change shape. A/B
./bench JavaVsJniWriteBenchmark.javaWriteand.javaWriteConcurrentat-f 3, before step 2 and after step 5.
Acceptance
./mvnw verifygreen after each step, integration (vortex-jni interop) included.- Byte identity: write the real-file corpus (the CSV/Parquet files
RealFileScanBenchmarkuses) onmainand on the branch, and require matchingsha256per file. Run it once per variant: default,--compact, zone maps off, pre-core2026.08.0edition (legacyvortex.stats), cascading 0, and a tightglobalDictMaxRetainedBytesto force demotion.vortex importonly covers default and--compact, so the other variants need a throwaway harness overWriteOptions(not committed). Output is executor-independent by design, so any diff is a bug. - No CHANGELOG entry (internal refactor, nothing user-visible). No AGENTS.md change (the module map lists public types only).
Activity
refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink…
refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink…
refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink…
refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink…
refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink…