Skip to content

feat(writer): write vortex.union (declared in Rust's unstable2026.10.0 edition) #660

Open

Description

@dfa1

Write side of #442. #442 deferred the writer because vortex.union was in no Rust edition ("Revisit when an edition declares it"). That is no longer true: Rust's unstable2026.10.0 edition (vortex/editions/unstable/unstable2026.10.0.toml, vortex-edition/src/declarations/unstable/mod.rs) lists vortex.union. docs/compatibility.md (lines 39, 150) still says "in no edition" and needs correcting as part of this.

Scope

  • UnionEncodingEncoder in writer.encode, registered in WriteRegistry.Builder#registerDefaults(): type ids (U8, nullable iff the union is) + one row-aligned child per variant, no metadata, no buffers (Rust rejects either). Layout per Support DType::Union and the vortex.union array (read side) #442.
  • Cascading: compress type_ids and each child independently and rebuild the union, as vortex-compressor/src/compressor/cascade.rs does.
  • Edition gating: Rust now has an unstable family; EditionFamily is CORE/PREVIEW/ZSTD. Decide whether to add it (ADR 0023 says closed enum mirroring Rust's families, so probably yes) and make a default write never emit vortex.union.
  • Update docs/compatibility.md, AGENTS.md if the edition family changes, CHANGELOG.

Testing (vortex-jni cannot help: vortex-arrow bails on DataType::Union)

Ideas, cheapest first:

  1. Rust-native cargo harness (as for the Pco encoder): a small Rust program linking vortex-array/vortex-file that (a) reads a Java-written file and dumps the array tree + values, and (b) builds a UnionArray and writes it with the unstable edition, so Java reads real Rust output. This also gives the Support DType::Union and the vortex.union array (read side) #442 decoder its first real-producer coverage.
  2. Rust-side vtx/vortex-tui browse on a Java-written file, as a quick manual smoke check.
  3. Byte-compare node trees: write the same logical union through the Rust harness and Java, diff the ArrayNode trees (the replicate-JNI-on-size-gaps approach).
  4. Reuse Rust's own test vectors in vortex-array/src/arrays/union/tests as unit fixtures.
  5. Keep the hand-built reader tests from Support DType::Union and the vortex.union array (read side) #442; add random unions via a seeded @MethodSource (variant counts 1..256, non-consecutive type ids, null tags, nested union children).

Open question: is the unstable edition's vortex.union wire layout final? It makes no stability promise, so changes may follow.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions