Skip to content

Nested struct columns are written as one flat array, not Rust's struct layout #724

Closed

Description

@dfa1

Rust writes a struct column as a vortex.struct layout: each field is its own column, chunked and compressed on its own, with a validity child when the struct is nullable. vortex-java's LayoutTree uses STRUCT only for the file root, so a nested struct column is one flat struct array (under vortex.masked when nullable).

Found on Raincloud mnist: image (struct<bytes: binary, path: utf8>) is 2.2% bigger than Rust's (18.18 MB vs 17.80 MB from vortex-jni 0.91). Rust's layout also lets a reader project or prune one field without decoding the others.

Fix: recurse into struct fields at the layout level, with per-field repartition and zone maps and a validity child. The reader already reads these layouts in Rust-written files. Largest of the three Raincloud size gaps; depends on the nested repartition issue for per-field chunking.

Test first: a vortex-jni cross-check writing a nullable struct column, asserting Java's layout tree matches Rust's.

Activity

  1. added
    bugSomething isn't working
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions