Repository navigation
Drop null patch elements when sparse lists share their elements - #11182
Conversation
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 4, 2026
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 4, 2026
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 1↑ 1↓)
|
commented
Nov 4, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.983x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.974x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.986x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.013x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.004x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.974x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.011x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.986x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.021x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.007x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 1↑ 2↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.047x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.990x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.066x ➖, 0↑ 1↓)
|
commented
Nov 4, 2026
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.008x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.990x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: JOB on NVME 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 2↑ 2↓)
duckdb / duckdb / ns (0.999x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.008x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 1↓)
duckdb / duckdb / ns (1.008x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 4↑ 3↓)
datafusion / vortex-compact / ns (1.095x ➖, 1↑ 4↓)
datafusion / parquet / ns (1.089x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed / ns (1.186x ➖, 0↑ 4↓)
duckdb / vortex-compact / ns (0.964x ➖, 3↑ 3↓)
duckdb / parquet / ns (0.975x ➖, 1↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.011x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.053x ➖, 1↑ 3↓)
datafusion / vortex-compact / ns (0.984x ➖, 2↑ 3↓)
datafusion / parquet / ns (1.081x ➖, 2↑ 4↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.049x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.160x ➖, 0↑ 5↓)
|
Summary
Since #11173, sparse lists with a null or empty fill reuse the patch lists' elements in their output. When the patch lists are already zero-copy to a list,
MakeExactonly trims the ends, so null patch lists keep their elements. Those elements are exported and written.This PR uses
MakeExactWithEmptyNullson the null or empty fill path so that null patches are lists of length 0.Changes
MakeExactWithEmptyNulls.empty_fill_reuses_elementstest now uses a null patch with no elements.