Skip to content

Nested columns are not repartitioned: one chunk per batch instead of Rust's ~1 MB chunks #723

Open

Description

@dfa1

Repartitioner.supports covers Primitive, Bool, Decimal, Utf8 and Binary; struct, list, fixed-size-list, map and variant columns are written one chunk per input batch. Rust's RepartitionStrategy coalesces every column, nested ones included, into ~1 MB chunks of 8192-row multiples.

Found on Raincloud coig: conversations (list<struct<question: utf8, answer: utf8>>) is written as 3 chunks of 65536 rows (the Parquet import's batches), where Rust writes 22 chunks of 8192 rows. The column is 2.6% bigger than Rust's (63.74 MB vs 62.13 MB from vortex-jni 0.91), and a row-range or pruned read touches 8x more rows per chunk.

Fix: byte sizing and head/tail slicing for StructData, ListData and FixedSizeListData carriers, then admit them in supports.

Test first: extend RepartitionInteropIntegrationTest with list<struct<utf8>> and struct columns, comparing chunk row counts and sizes with vortex-jni.

Activity

  1. added
    bugSomething isn't working
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions