Skip to content

Flatten struct children inside DuckDB list exports - #11190

Closed
mprammer wants to merge 1 commit into
vortex-data:developfrom
mprammer:mp/duckdb-list-struct-flatten
Closed

mprammer wants to merge 1 commit into
vortex-data:developfrom
mprammer:mp/duckdb-list-struct-flatten

Conversation

@mprammer

@mprammer mprammer commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Reading a list<struct> column through the DuckDB extension fails with chunked DuckDB export range 0..N crosses chunk boundary at M as soon as the list's elements span more than one chunk, which is what the file reader produces for any such column larger than one row group. The list exporter copies all of its elements into one DuckDB vector in a single export(0, n) call and asks for a flattened element exporter, but the struct exporter ignored that request and built its children with flatten = false, so a chunked child became the chunk-aligned ChunkedExporter, which rejects a range crossing a chunk boundary.

Changes

flatten is now threaded through canonical::new_exporter and struct_::new_exporter to the struct's children, and through the two remaining paths that canonicalize and then build a child exporter (constant::new_exporter_with_validity and the flatten branch of the dictionary exporter). Top-level structs still use flatten = false, so their output and cost are unchanged; the only behaviour change is that struct children under a list, fixed-size list, or dictionary are canonicalized instead of erroring. test_export_list_of_structs_with_chunked_child reproduces the failure with a two-chunk child and a list that straddles the boundary.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Martin Prammer <martin@spiraldb.com>
@mprammer
mprammer requested a review from myrrc November 4, 2026 04:22
@myrrc myrrc added the action/bench-sql-extended Run all regular SQL benchmarks with full PR target coverage label Nov 4, 2026
@myrrc

myrrc commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

I don't think we need to flatten everything

@robert3005

Copy link
Copy Markdown
Contributor

we have fixed it in #11204

@robert3005 robert3005 closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action/bench-sql-extended Run all regular SQL benchmarks with full PR target coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants