Repository navigation
feat: support scan planning and pushdown for Java table providers #119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
wirybeaver
wants to merge
2
commits into
apache:main
Choose a base branch
from
wirybeaver:feat/java-table-scan-pushdown
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+860
−82
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
43 changes: 43 additions & 0 deletions
43
core/src/main/java/org/apache/datafusion/ExecutionPlan.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.datafusion; | ||
|
|
||
| import org.apache.arrow.memory.BufferAllocator; | ||
| import org.apache.arrow.vector.ipc.ArrowReader; | ||
|
|
||
| /** | ||
| * A bounded Java-backed leaf plan returned by {@link TableProvider#scanWithArgs(ScanArgs)}. | ||
| * | ||
| * <p>Constructing the scan must not read source rows. Each execution opens a fresh independent | ||
| * reader whose schema is either the registered full schema or its requested projection. Reader | ||
| * buffers must use the supplied allocator or a child of it. DataFusion owns and closes the reader. | ||
| * | ||
| * <p>This is a Java-backed scan, not an API for arbitrary child-consuming physical operators. | ||
| */ | ||
| @FunctionalInterface | ||
| public interface ExecutionPlan { | ||
| /** | ||
| * Open an independent stream for a partition during physical execution. | ||
| * | ||
| * <p>The current adapter executes only partition {@code 0}; the partition argument is explicit so | ||
| * supporting multi-partition plans later does not require changing this method's signature. | ||
| */ | ||
| ArrowReader execute(int partition, BufferAllocator allocator); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.datafusion; | ||
|
|
||
| import java.util.List; | ||
| import java.util.Optional; | ||
| import java.util.OptionalLong; | ||
|
|
||
| import org.apache.datafusion.protobuf.LogicalExprNode; | ||
|
|
||
| /** | ||
| * A physical scan request using DataFusion's projection, expression and limit semantics. | ||
| * | ||
| * <p>Projection indices refer to the registered table schema. An absent projection means all | ||
| * columns; a present empty list means no columns (for example, {@code COUNT(*)}), not no rows. | ||
| * Filters use the bundled DataFusion version's expression protobufs, not SQL strings. Their | ||
| * generated types and expression shapes may change when the binding upgrades DataFusion. | ||
| * | ||
| * <p>The optional limit is the value supplied by DataFusion's planner. Sources may return more | ||
| * rows; they must not return fewer than requested when that many matching rows are available. | ||
| * DataFusion retains responsibility for deciding whether a limit can safely cross a residual | ||
| * filter. Limits outside the non-negative Java {@code long} range are not forwarded; DataFusion | ||
| * still applies the query's limit. | ||
| */ | ||
| public record ScanArgs( | ||
| Optional<List<Integer>> projection, List<LogicalExprNode> filters, OptionalLong limit) { | ||
| public ScanArgs { | ||
| projection = projection.map(List::copyOf); | ||
| filters = List.copyOf(filters); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This exposes the generated
LogicalExprNodeas public API. That type tracks DataFusion'sdatafusion.protofor the bundled version, and its package is excluded from our Javadoc. Pushdown code that matches onBinaryExprops and literal variants, like the test'sSelectiveSource, could break when we bump DataFusion. I think passing the proto is a reasonable first step, but should we say intable-provider.mdthat the expression format follows DataFusion's proto and may change between DataFusion versions? It would be good to agree on that compatibility story before this ships in a release.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my own java bindings I'd original made it a goal for the expression APIs (logical and physical) to use their own POJOs of sealed classes and records where it made sense, with the hope of the API feeling friendlier, and hiding the generated classes unless the caller wanted to pass them. I backed off voicing that goal when I saw the protos start to get exposed in this repo and I couldn't really explain why I thought it was important. You do highlight one possible compatibility issue that swings me back in the other direction.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, here is my working draft of the "public API" agent instructions that I mentioned wanting to do many months ago. I have larger ambitions for it but tried to slim it down for what is realistic right now.
If that seems directionally correct and only require small tweaks, I'd be happy to open a proper PR. There are some minor divergences from the rules there which I could open up follow-up PRs for.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. The guide and
ScanArgsJavadoc explain thatLogicalExprNodefollows the bundled DataFusion version; providers should decline unfamiliar forms withUNSUPPORTEDand retest on upgrade. I’d keep a stable Java expression model as a separate API discussion.