Skip to content

refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink, LayoutTree, GlobalDictWriter and ZoneMapWriter (#663) - #683

Merged
dfa1 merged 1 commit into
mainfrom
refactor-vortex-writer-663
Nov 7, 2026
Merged

dfa1 merged 1 commit into
mainfrom
refactor-vortex-writer-663

Conversation

@dfa1

@dfa1 dfa1 commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Closes #663.

VortexWriter went from 1772 to 396 lines. It keeps the public factories, schema validation, writeChunk, column chunking and repartition, and the close() orchestration. Five package-private collaborators take the rest:

  • SegmentEncoder: candidate lists and registries, the edition guard, ColumnEncoding overrides, the dict probe, per-segment encode and stats. It's immutable (the caller's encoder list is copied), and it's the only writer state the executor tasks touch.
  • SegmentSink: the channel, the ordered window of segments being encoded, segment and encoding-id bookkeeping, the array/stats FlatBuffers, footer, postscript and trailer.
  • LayoutTree: the layout tree and the footer's layout_specs list it indexes into. It's the only place FbsLayout nodes are built.
  • GlobalDictWriter: first-batch detection, ingest, budget eviction and demotion (replayed through a ChunkAppender callback), and the flush at close.
  • ZoneMapWriter: the zoned accumulators, and the vortex.zoned and legacy vortex.stats flush.

RowCount.of (in writer.encode) replaces three copies of the row-count logic: VortexWriter.arrayLength, the switch in CascadingCompressor and ZoneAccumulator's reflective length.

Removed along the way: dictCandidates (it was always dictStates.keySet()), the retained-bytes cache (always equal to the live sum), a duplicated CompletionException unwrap, inline fully-qualified names, stray docs, wrapped ternaries and inline field initializers (every field is now assigned in its constructor).

Verification

  • Byte identity: 12 Raincloud Parquet files × 8 WriteOptions variants (default, compact, zone maps off, legacy edition, no editions, forced dict demotion, cascading 0, global dict off). All 96 outputs have the same sha256 as on main.
  • ./mvnw verify -pl writer -am: 2576 tests, checkstyle and javadoc clean.
  • ./mvnw verify -pl integration -am (vortex-jni interop): see the latest comment for the run on the final commit.

Found while testing, already present on main and not fixed here: WriteOptions.cascading(0) fails on 4 of the 12 files (vortex.masked: no inner encoding for Bool/Struct, Argument is not an array).

🤖 Generated with Claude Code

@dfa1
dfa1 force-pushed the refactor-vortex-writer-663 branch 2 times, most recently from 291e50c to bdbd549 Compare November 7, 2026 07:03
@dfa1 dfa1 changed the title refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink, GlobalDictWriter and ZoneMapWriter (#663) refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink, LayoutTree, GlobalDictWriter and ZoneMapWriter (#663) Nov 7, 2026
@dfa1
dfa1 force-pushed the refactor-vortex-writer-663 branch from bdbd549 to aec4df1 Compare November 7, 2026 07:04
…, LayoutTree, GlobalDictWriter and ZoneMapWriter (#663)

VortexWriter was 1772 lines with ~40 fields and no grouping. It keeps the
public factories, schema validation, writeChunk, column chunking and
repartition, and the close() orchestration (396 lines); five
package-private collaborators take the rest:

- SegmentEncoder: candidate lists and registries, the edition guard,
  ColumnEncoding overrides, the dict probe, per-segment encode + stats.
  Immutable (the caller's encoder list is copied), and the only state
  the executor tasks touch.
- SegmentSink: channel, ordered in-flight window, segment and
  encoding-id bookkeeping, array/stats FlatBuffers, footer, postscript,
  trailer.
- LayoutTree: the layout tree and the footer's layout_specs it indexes;
  the only place FbsLayout nodes are built.
- GlobalDictWriter: first-batch detection, ingest, budget eviction and
  demotion (back through a ChunkAppender callback), flush.
- ZoneMapWriter: zoned accumulators, vortex.zoned and legacy
  vortex.stats flush.

RowCount.of (writer.encode) replaces VortexWriter.arrayLength,
CascadingCompressor's switch and ZoneAccumulator's reflective length.
Also dropped redundant state (dictCandidates was dictStates.keySet();
the retained-bytes cache equalled the live sum), one CompletionException
unwrap, inline FQNs, stray docs, wrapped ternaries and inline field
initializers (every field is assigned in its constructor).

No behavior change: 96 files (12 Raincloud Parquet files x 8 option sets
incl. legacy edition, cascading 0, forced demotion) are byte-identical
to main; writer and integration (vortex-jni) suites pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 force-pushed the refactor-vortex-writer-663 branch from aec4df1 to edde7c8 Compare November 7, 2026 07:05
@dfa1
dfa1 merged commit 11c3fe3 into main Nov 7, 2026
6 checks passed
@dfa1
dfa1 deleted the refactor-vortex-writer-663 branch November 7, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: VortexWriter must be simplified

1 participant