Repository navigation
Conversation
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.978x ➖, 2↑ 1↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.975x ➖, 1↑ 0↓)
duckdb / duckdb / ns (1.018x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.847x ✅, 5↑ 0↓)
datafusion / vortex-compact / ns (0.869x ✅, 5↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 2↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.065x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (0.898x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.871x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed / ns (0.900x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.925x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.968x ➖, 1↑ 2↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: JOB on NVME 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 1↑ 2↓)
duckdb / duckdb / ns (1.008x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.889x ✅, 39↑ 5↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 1↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 2↑ 3↓)
datafusion / vortex-compact / ns (1.004x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.011x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 1↓)
duckdb / duckdb / ns (1.010x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 1↓)
duckdb / duckdb / ns (1.004x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.052x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.060x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.783x ➖, 5↑ 0↓)
datafusion / vortex-compact / ns (0.866x ➖, 4↑ 2↓)
datafusion / parquet / ns (0.988x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.857x ➖, 3↑ 0↓)
duckdb / vortex-compact / ns (1.010x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.845x ➖, 3↑ 0↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.796x ➖, 6↑ 0↓)
datafusion / vortex-compact / ns (0.921x ➖, 3↑ 2↓)
datafusion / parquet / ns (0.860x ➖, 4↑ 1↓)
duckdb / vortex-file-compressed / ns (0.948x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (0.981x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.928x ➖, 2↑ 0↓)
|
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
0253169 to
e909ee5
Compare
Summary
Several distinct
list_transformlambdas over one list, or a lambda beside the list read whole, go into one elements read. The elements reader packs one value per lambda, plus the raw element when needed, and the lists come back as a struct of lists sharing the offsets and validity. Over the whole stack, the microbench geomean of 112 cases is 0.65x of develop.Changes
split_list_transform(list/expr.rs): packs the bodies as"0","1", ... and rewrites each use to its list.lists_over(list/reader.rs): one list per pack field, extracted lazily from the packed elements.Testing
list_transform_reads_only_what_its_body_uses::two_lambdasnow reads onlyaandf.random_list_uses_match_memorycovers two lambdas besidelist_sum,list_length,is_null, and the whole list.Follow-up
list_transformforis_nullandlist_length.