Skip to content

refactor: VortexWriter must be simplified #663

Closed
1 of 1 issue completed
Closed

Description

@dfa1

VortexWriter is 1712 lines with ~40 mutable fields and no grouping: encoder selection, the async segment pipeline, file I/O, the global dictionary, zone maps, repartitioning and layout-tree building all share one class and one namespace. The only thing keeping the executor thread-safe is a comment ("touches only immutable writer state").

Where the state actually clusters

Cluster Fields Methods ~Lines
Encoder selection (immutable, runs on the executor) encodings, defaultRegistry, cascadeCodecs/cascadeRegistry, probeCodecs/probeRegistry, editionAllowed/editionExcluded, columnEncoders buildCascadeCodecs, editionAllowed/Excluded/HasZonedLayout, columnEncoder, zoneTableEncoder, resolveOverride, findEncoder, nullableCapableEncoder, encodeSegment, probeCompressesToDict ~400
Segment sink (ordered I/O + footer) channel, bytesWritten, segs, encodingIdx, pending, submittedSegments writeSegment×3, drainSegments, discardPending, writeEncoded, registerEncodingIds, write×2, writePadding, bufferPaddings, buildArrayFlatBuffer, buildStats, buildArrayNodeFlatBuffer, buildFooter, buildPostscript, trailer ~350
Global dictionary dictCandidates, dictStates, dictColRefs, dictRetainedBytes, dictRetainedTotal, dictRetainedBudget, firstChunkSeen candidate detection (inline in writeChunk), evictLargest…, demoteDictColumn, flushDictColumns, writeGlobalDict*, dictPool, writeGlobalDictCodes, buildDictColLayout, buildDictLayoutMetaBytes ~300
Zone maps zoneMaps, zoneAccumulators, zoneTails, zonedLayout flushZoneMaps, joinZoneTails, emitZoneMap, wrapZoneMap ~200
Columns (what stays) schema, options, colChunks, repartitioners, pieceSizes create×3, ctor validation, writeChunk, appendChunk, writeDataChunk, flushRepartitioners, buildLayout, close ~450

Redundant state found while mapping:

  • dictCandidates is always dictStates.keySet() (added and removed together).
  • dictRetainedBytes + dictRetainedTotal cache DictColumnState#retainedBytes(), which only changes when that column ingests: the total is a sum over the candidate columns' dictionaries (done in c06eb2ee).
  • drainSegments and joinZoneTails duplicate the CompletionException unwrap.

Plan

Pure refactor: no public API change, byte-identical output. New types are package-private final classes in io.github.dfa1.vortex.writer, the same way DictColumnState, ZoneAccumulator and Repartitioner already are. One commit per step, each green on ./mvnw verify.

  1. Cleanup, no moves. Delete the redundant state above. Add one helper for the completion unwrap. Use imports instead of the inline FQNs (io.github.dfa1.vortex.core.fbs.FbsArray*, …encode.NullableData, java.util.HashSet). Fix the duplicated create javadoc, the two docs merged on editionHasZonedLayout, and the orphan comments ("Indices into layout_specs", "Base cascade codec list"). Move countNulls next to NullableData.
  2. Extract SegmentEncoder. This is the immutable encoder-selection cluster, built once in the constructor; the executor lambdas capture it instead of this, so thread safety comes from the types rather than a comment. Watch the ordering: columnEncoder's anonymous class currently reads editionExcluded lazily, and that field is assigned after columnEncoders is built. Compute the exclusions first and pass them in explicitly.
  3. Extract SegmentSink. It owns the channel, the in-flight window, segment and encoding-id bookkeeping, buffer and stats FlatBuffers, the footer, postscript and trailer. API: int submit(Supplier<EncodedSegment>), drain(), stats(segIdx) (for the legacy zone map), and finish(dtypeBuf, layoutBuf). The edition backstop in registerEncodingIds takes the allowed set.
  4. Extract GlobalDictWriter. It covers detection on the first batch, ingest, eviction and demotion, the flush at close, and its own layout subtree (buildDictColLayout, metadata bytes). Demotion replays chunks into the column path through a small ChunkSink callback (appendChunk), not a back-reference to the writer. writeChunk asks it isCandidate(col).
  5. Extract ZoneMapWriter. It feeds the zoned accumulators per batch, flushes both the zoned and the legacy vortex.stats maps, and does wrapZoneMap. The legacy path reads chunk stats from SegmentSink#stats and dict chunk stats from GlobalDictWriter.

Target: VortexWriter at about 450 lines. It keeps the public factories, schema validation, writeChunk (adapt, row-count check, extension routing, piece sizing, dispatch), column chunking and repartition, buildLayout, and the close() orchestration. arrayLength stays a package-private static there, since every collaborator uses it.

Out of scope: any behavior change, the MAX_IN_FLIGHT_SEGMENTS knob, and the concurrent-scan work.

Risks

  • close() order is load-bearing: repartitioner flush, then dict flush, then zone maps, then drain, then footer. Global-dict demotion during writeChunk interleaves segment indexes. Any reordering changes the segment layout of the file.
  • Hidden ordering in the constructor: see step 2.
  • Perf: the hot loops are in the encoders, not here, but the executor closures change shape. A/B ./bench JavaVsJniWriteBenchmark.javaWrite and .javaWriteConcurrent at -f 3, before step 2 and after step 5.

Acceptance

  • ./mvnw verify green after each step, integration (vortex-jni interop) included.
  • Byte identity: write the real-file corpus (the CSV/Parquet files RealFileScanBenchmark uses) on main and on the branch, and require matching sha256 per file. Run it once per variant: default, --compact, zone maps off, pre-core2026.08.0 edition (legacy vortex.stats), cascading 0, and a tight globalDictMaxRetainedBytes to force demotion. vortex import only covers default and --compact, so the other variants need a throwaway harness over WriteOptions (not committed). Output is executor-independent by design, so any diff is a bug.
  • No CHANGELOG entry (internal refactor, nothing user-visible). No AGENTS.md change (the module map lists public types only).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions