Repository navigation
Push DataFusion dynamic filters into V2 scans - #11215
Conversation
Inline roots shared one session, and the run clears a root's session when the root retires. Once the first root finished, the remaining roots waited for reads that were never issued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Evaluate TopK and hash join dynamic filters in the V2 file scan, so zones and rows are dropped before the engine sees them. The producing operator keeps enforcing them. V1 scans keep them out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's hash table lookup has no Vortex equivalent, so one unconvertible conjunct turned the whole filter into a constant true and its bounds were lost. Convert each conjunct alone and keep the ones that convert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's dynamic filter bounds its probe keys with key >= min AND key <= max. Each comparison decoded the key column again, and the scan's between rewrite only ran on static filters, so the bounds now merge into one between when the filter is read. TPC-DS SF10, 10 join queries against the previous commit: geomean 0.972, Q69 and Q10 0.93, none slower than 1.007. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A scan of at most one filtered chunk evaluates its conjuncts on that chunk next, so pruning its zones first saves little and delays the chunk, and pruning a dynamic conjunct proves its current predicate over the zones again. DataFusion splits small join probes into many such scans. Larger scans keep pruning their dynamic conjuncts: skipping it for every scan made TPC-DS SF10 Q82 and Q37 3.4-3.7x slower. TPC-DS, all 99 queries against the previous commit: SF1 geomean 0.97, SF10 0.99 on best runs and 1.01 on medians; the SF10 outliers were flat when rerun. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…es change A dynamic conjunct whose bound never prunes still costs a zone-pruning pass before each scan and an evaluation per chunk. With `ScanBuilder::with_skip_unselective_dynamic`, a conjunct containing dynamic expressions that kept at least 99% of its input rows on four consecutive chunks at one dynamic version is switched off: its zone pruning and its evaluation are skipped until its dynamic values change. The scan can then return rows the conjunct would reject, so this is opt-in. Only the DataFusion persistent opener enables it, because DataFusion's TopK and hash-join dynamic filters are pushed down inexactly and the producing operator applies them again. DuckDB and other V2 scans are unchanged. Cached filter plans are reused only by scans with the same setting. `find_between` only merges comparisons against literals, so a switched-off conjunct never carries a static restriction with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBZYz5ae2yLro3TPmH Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A chunk announced the reads of every conjunct before evaluating any, so a switched-off conjunct still had its column fetched. The scan test now checks that a switched-off scan reads no more than an unfiltered one, and tightens the bound enough that the conjunct stays switched on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBZYz5ae2yLro3TPmH Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Previous Runs (17)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.057x ➖, 0↑ 2↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.050x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.023x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.959x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.009x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.070x ➖, 2↑ 30↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.986x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.965x ➖, 2↑ 1↓)
datafusion / parquet / ns (0.994x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.901x ➖, 3↑ 0↓)
duckdb / parquet / ns (1.025x ➖, 0↑ 1↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.905x ➖, 29↑ 6↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.993x ➖, 1↑ 1↓)
|
`filter_run_end_primitive` counted every run's range of the mask from scratch. On columns with short runs, such as TPC-DS `ss_ticket_number`, that is one popcount call per run, and it took most of the extra CPU of DataFusion scans that apply dynamic filters (q73 SF1: 2.5 s of 2.8 s extra over 150 iterations). Run ends increase, so walk the mask's words once with a running rank instead. `run_end_filter` bench, median before -> after: 4K rows run 16: 1.28 us -> 0.52 us; 16K rows run 16: 4.75 us -> 1.81 us; 64K rows run 4: 65.0 us -> 24.0 us; 64K rows run 256: 1.79 us -> 0.86 us. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.014x ➖, 0↑ 1↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.030x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.992x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.933x ➖, 2↑ 0↓)
datafusion / vortex-compact / ns (0.943x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.080x ➖, 0↑ 3↓)
duckdb / vortex-compact / ns (1.091x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.970x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.061x ➖, 3↑ 27↓)
datafusion / vortex-compact / ns (1.120x ❌, 1↑ 45↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.001x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.022x ➖, 2↑ 4↓)
datafusion / vortex-compact / ns (0.941x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.976x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (1.045x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (1.038x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.049x ➖, 0↑ 2↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.968x ➖, 2↑ 0↓)
datafusion / vortex-compact / ns (0.955x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.021x ➖, 0↑ 3↓)
duckdb / vortex-compact / ns (1.018x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.998x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.984x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.026x ➖, 0↑ 1↓)
|
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / parquet / ns (0.998x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.989x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Merging this PR will improve performance by 22.33%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.183x ➖, 0↑ 4↓)
random-access / parquet / ns (1.088x ➖, 0↑ 1↓)
random-access / lance / ns (1.051x ➖, 0↑ 1↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.979x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.286x ➖, 1↑ 9↓)
datafusion / vortex-compact / ns (1.243x ➖, 1↑ 9↓)
datafusion / parquet / ns (0.969x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.048x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.111x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.099x ➖, 0↑ 6↓)
random-access / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
random-access / parquet / ns (1.002x ➖, 0↑ 0↓)
random-access / lance / ns (1.002x ➖, 0↑ 0↓)
random-access / litdata / ns (1.007x ➖, 0↑ 0↓)
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Fold the commits of the V2 dynamic filter branch into one change on top of develop, which pushed DataFusion dynamic filters into V2 scans in #11215: - Push DataFusion dynamic filters into V2 scans as switched conjuncts - Drop dynamic filter parts that file statistics show keep every row - Filter run-end arrays by mask rank and by set bits for sparse masks - Log the native filter of each Vortex scan in DataFusion - Share switch measurements across files and push routed join filters - Route rows of partitioned join filters through a table of branches - Convert Scan API dynamic filters conjunct by conjunct - Prune with the bounds a multi-column dynamic filter implies The switch of each optional conjunct replaces develop's skipping of unselective dynamic conjuncts, so ScanBuilder::with_skip_unselective_dynamic and the dynamic filter path through the V2 scan predicate are removed. The dynamic filters reach the scan through the file pruning predicate for V1 and V2 scans. V2 scans keep develop's skip of dynamic pruning in scans of one filtered chunk, and also skip the pruning of a paused switch. The run-end filter keeps develop's rank walk with POPCNT dispatch and adds the walk over set bits for masks that keep fewer rows than there are runs. The Scan API dynamic filter keeps develop's merge of key bounds into a between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
Fold the commits of the V2 dynamic filter branch into one change on top of develop, which pushed DataFusion dynamic filters into V2 scans in #11215: - Push DataFusion dynamic filters into V2 scans as switched conjuncts - Drop dynamic filter parts that file statistics show keep every row - Filter run-end arrays by mask rank and by set bits for sparse masks - Log the native filter of each Vortex scan in DataFusion - Share switch measurements across files and push routed join filters - Route rows of partitioned join filters through a table of branches - Convert Scan API dynamic filters conjunct by conjunct - Prune with the bounds a multi-column dynamic filter implies The switch of each optional conjunct replaces develop's skipping of unselective dynamic conjuncts, so ScanBuilder::with_skip_unselective_dynamic and the dynamic filter path through the V2 scan predicate are removed. The dynamic filters reach the scan through the file pruning predicate for V1 and V2 scans. V2 scans keep develop's skip of dynamic pruning in scans of one filtered chunk, and also skip the pruning of a paused switch. The run-end filter keeps develop's rank walk with POPCNT dispatch and adds the walk over set bits for masks that keep fewer rows than there are runs. The Scan API dynamic filter keeps develop's merge of key bounds into a between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
Very interesting! This is essentially what we are planning on doing in DataFusion as well. Just blocked on review capacity... |
develop now scans with the V2 scan by default and already pushes DataFusion dynamic filters into it (#11215). The V1 scan changes on this branch are superseded, and the RunEnd files they touched moved and were rewritten, so this merge keeps develop's version of every file. The partitioned hash join handling is ported onto the V2 dynamic filter in the next commit. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01So4kxsqCY8WFG1NTMNJxKX
Summary
This is #11073 rebased onto
develop, plus an on/off switch for dynamic conjuncts. The switch is ported to Scan V2 from the unselective-skip commit in #10415.On TPC-DS SF1, CI showed #11073 alone with 38 of 99 DataFusion/Vortex queries slower and 2 faster. Most of that is the zone-pruning pass a dynamic filter adds before each scan. This PR checks whether switching off dynamic conjuncts that reject almost nothing recovers that, while keeping the SF10/SF20 gains. Everything is gated on
scan_v2, so the default V1 path is unchanged.Changes
between.v2::ScanBuilder::with_skip_unselective_dynamic. Only the DataFusion persistent opener enables it, because DataFusion pushes dynamic filters down inexactly (PushedDown::No) and the producing TopK or join applies them again. DuckDB is unchanged.find_betweenonly merges comparisons against literals, so a switched-off conjunct never carries a static restriction with it.Checks
cargo test -p vortex-layout -p vortex-datafusion: all passed (826 + 746 + 14 + 1, 2 ignored).cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warnings: clean.cargo +nightly-2026-09-10 fmt -p vortex-layout -p vortex-datafusion: clean.vortex-sqllogictest,vortex-arraytests.action/bench-sqlcovers TPC-DS SF1 for comparison with Push DataFusion dynamic filters into V2 scans #11073. SF10 is not in any PR preset.The switch needs 4 chunks to trip, so it may not help the smallest SF1 scans.
Generated by Claude Code