Repository navigation
duckdb: report distinct count based on min/max - #11130
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Merging this PR will improve performance by 10.83%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
768e129 to
0d976c5
Compare
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 3, 2026
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 3, 2026
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.028x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.991x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.012x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.006x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (1.024x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.991x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.018x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.982x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.160x ➖, 1↑ 3↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.129x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.126x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (1.138x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: JOB on NVME 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.006x ➖, 11↑ 11↓)
duckdb / parquet / ns (1.001x ➖, 2↑ 3↓)
duckdb / duckdb / ns (0.997x ➖, 1↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.004x ➖, 1↑ 1↓)
|
commented
Nov 3, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 12↑ 11↓)
duckdb / vortex-compact / ns (0.996x ➖, 13↑ 13↓)
duckdb / parquet / ns (1.003x ➖, 1↑ 1↓)
duckdb / duckdb / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.114x ➖, 1↑ 4↓)
datafusion / vortex-compact / ns (1.083x ➖, 2↑ 2↓)
datafusion / parquet / ns (1.077x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.975x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.985x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.065x ➖, 0↑ 2↓)
|
commented
Nov 3, 2026
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 0↓)
|
commented
Nov 3, 2026
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.213x ➖, 0↑ 4↓)
datafusion / vortex-compact / ns (1.027x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.045x ➖, 2↑ 3↓)
duckdb / vortex-file-compressed / ns (0.924x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (1.070x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.041x ➖, 0↑ 1↓)
|
DuckDB's JOIN planner overstates cardinality for table functions which push down
filters (duckdb/duckdb#28099). Hack this until upstream
PR is merged by reporting distinct count as MAX - MIN + 1 based on min/max
statistics.