Skip to content

Use the shared scan driver in the DataFusion file opener - #11611

Merged
gatesn merged 2 commits into
developfrom
ngates/datafusion-shared-scan-driver
Nov 19, 2026
Merged

gatesn merged 2 commits into
developfrom
ngates/datafusion-shared-scan-driver

Conversation

@gatesn

@gatesn gatesn commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Use the shared driver for the DataFusion V2 file opener and remove its duplicate scheduler. The opener still waits for scan preparation so failures are reported as file-open errors, and Arrow conversion still runs on the task that produces each batch.

When several chunks run on the same scheduler, such as with scan_concurrency=0, each chunk needs its own queue for pending reads. The shared driver used to give these chunks the same I/O session, so when one chunk finished and cleared the session, it could discard another chunk’s pending reads. Give each planner root its own session over the same file source, preserving the isolation DataFusion already had for inline chunks.

Stacked on #11610.
Part of #9062.

This change was prepared with assistance from OpenAI Codex.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants