You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support DType::Union and the vortex.union array (read side) #442
Reading a file whose schema contains a union fails today with VortexException("unsupported DType typeType=12") (docs/compatibility.md: DType::Union, Rust 0.71.0, ❌). Rust has both the dtype and its array encoding.
Rust reference (spiraldb/vortex develop)
DType: FlatBuffer table Union { names: [string]; dtypes: [DType]; type_ids: [byte]; nullable: bool; } (variant 12 of Type). Protobuf message Union { repeated string names = 1; repeated DType dtypes = 2; repeated int32 type_ids = 3; bool nullable = 4; }. Both schemas are already in our .fbs/.proto.
Validation (vortex-array/src/dtype/union.rs): names, dtypes and type_ids have equal length; at least 1 and at most 256 variants; type_ids are distinct (unsigned u8 tags, not necessarily consecutive, e.g. [0, 5, 7]); variant names are distinct (unlike Struct).
Array vortex.union (vortex-array/src/arrays/union): a canonical sparse union with no metadata and no buffers (Rust rejects either). Children are [type_ids, child_0 … child_{N-1}]: type_ids is U8, nullable iff the union is nullable, and each variant child is row-aligned with the full array length. Row i = the child whose tag equals type_ids[i], read at row i. A null tag means a null row.
Compressor (vortex-compressor/src/compressor/cascade.rs): compresses type_ids and each child independently and rebuilds the union, stored inside ordinary layouts (there is no union layout).
Scope (parity, read side)
DType.Union variant (sealed, with Rust's validation), parsed from the FlatBuffer (PostscriptParser) and the proto DType.
UnionEncodingDecoder plus a reader array type that exposes tag and validity per row and the variant children. Slicing, scans and CSV export (render the selected child's value) need to handle it.
Interop tests: Rust's Arrow conversion bails on unions (vortex-arrow/src/dtype.rs: DType::Union(..) => vortex_bail!("TODO(connor)[Union]: unimplemented")), so vortex-jni can neither write nor read them, and no Rust compat fixture contains one. Tests must be hand-built from the reference above until a Rust producer exists.
Activity
feat(reader): read DType::Union and the vortex.union sparse union arr…
docs(readme): list Vorticity (.NET) among Vortex implementations
docs(readme): list Vorticity (.NET) among Vortex implementations