Skip to content

Support DType::Union and the vortex.union array (read side) #442

Closed

Description

@dfa1

Reading a file whose schema contains a union fails today with VortexException("unsupported DType typeType=12") (docs/compatibility.md: DType::Union, Rust 0.71.0, ❌). Rust has both the dtype and its array encoding.

Rust reference (spiraldb/vortex develop)

  • DType: FlatBuffer table Union { names: [string]; dtypes: [DType]; type_ids: [byte]; nullable: bool; } (variant 12 of Type). Protobuf message Union { repeated string names = 1; repeated DType dtypes = 2; repeated int32 type_ids = 3; bool nullable = 4; }. Both schemas are already in our .fbs/.proto.
  • Validation (vortex-array/src/dtype/union.rs): names, dtypes and type_ids have equal length; at least 1 and at most 256 variants; type_ids are distinct (unsigned u8 tags, not necessarily consecutive, e.g. [0, 5, 7]); variant names are distinct (unlike Struct).
  • Array vortex.union (vortex-array/src/arrays/union): a canonical sparse union with no metadata and no buffers (Rust rejects either). Children are [type_ids, child_0 … child_{N-1}]: type_ids is U8, nullable iff the union is nullable, and each variant child is row-aligned with the full array length. Row i = the child whose tag equals type_ids[i], read at row i. A null tag means a null row.
  • Compressor (vortex-compressor/src/compressor/cascade.rs): compresses type_ids and each child independently and rebuilds the union, stored inside ordinary layouts (there is no union layout).

Scope (parity, read side)

  • DType.Union variant (sealed, with Rust's validation), parsed from the FlatBuffer (PostscriptParser) and the proto DType.
  • UnionEncodingDecoder plus a reader array type that exposes tag and validity per row and the variant children. Slicing, scans and CSV export (render the selected child's value) need to handle it.
  • vortex.union is a well-known EncodingId.

Not in scope yet

  • Writer: vortex.union is in no Rust edition, so Rust's edition-gated writer does not emit it. Emitting it here would go beyond Rust. Revisit when an edition declares it (see Edition catalog has drifted from Rust's vortex-edition declarations #441).
  • Interop tests: Rust's Arrow conversion bails on unions (vortex-arrow/src/dtype.rs: DType::Union(..) => vortex_bail!("TODO(connor)[Union]: unimplemented")), so vortex-jni can neither write nor read them, and no Rust compat fixture contains one. Tests must be hand-built from the reference above until a Rust producer exists.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions