Skip to content

Export DuckDB chunked arrays across chunk boundaries - #11204

Merged
myrrc merged 2 commits into
developfrom
rk/duckdb-chunked-cross-boundary-export
Nov 5, 2026
Merged

myrrc merged 2 commits into
developfrom
rk/duckdb-chunked-cross-boundary-export

Conversation

@robert3005

@robert3005 robert3005 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

When you read a list<struct> column through the DuckDB extension, the read fails with chunked DuckDB export range 0..N crosses chunk boundary at M as soon as the list elements span more than one chunk. The file reader gives this shape for any such column that is larger than one row group. The list exporter exports all its elements in one call, and ChunkedExporter only accepted ranges inside one chunk.

ChunkedExporter now accepts any range:

  • Range inside one chunk: that chunk's exporter exports it, as before, and keeps its encoding.
  • All chunks are dictionaries over the same values: the codes are joined and the range becomes one reuse_dictionary vector over the cached ReusableDict. Nothing is copied, and DuckDB keeps the dictionary ID, so its dictionary caches still apply.
  • Anything else: the range is sliced and canonicalized in Vortex. This joins string views without copying string data, then exports the result.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix ext/duckdb Relates to the DuckDB integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants