Repository navigation
perf(layout): evaluate plain struct field projections directly - #10372
joseph-isaacs wants to merge 2 commits into
Conversation
Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (1.002x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.015x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.012x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (0.970x ➖, 2↑ 0↓)
duckdb / parquet / ns (1.048x ➖, 0↑ 2↓)
File Size Changes (200 files changed, -0.0% overall, 98↑ 102↓)
Totals:
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.206x ➖, 0↑ 4↓)
datafusion / vortex-compact / ns (1.119x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.434x ❌, 0↑ 6↓)
duckdb / vortex-file-compressed / ns (0.897x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.100x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.111x ➖, 0↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 2↑ 1↓)
duckdb / vortex-compact / ns (1.001x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 3↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.001x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 3↑ 3↓)
duckdb / vortex-compact / ns (1.002x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.006x ➖, 2↑ 3↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.950x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.995x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 1↑ 3↓)
duckdb / vortex-compact / ns (0.968x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.944x ➖, 7↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.963x ➖, 5↑ 0↓)
vortex / parquet / ns (0.998x ➖, 0↑ 1↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.972x ➖, 4↑ 1↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.121x ➖, 3↑ 6↓)
datafusion / vortex-compact / ns (0.921x ➖, 3↑ 1↓)
datafusion / parquet / ns (1.139x ➖, 0↑ 5↓)
duckdb / vortex-file-compressed / ns (1.023x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (1.010x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.017x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.247x ➖, 0↑ 8↓)
datafusion / parquet / ns (1.121x ➖, 1↑ 5↓)
duckdb / vortex-compact / ns (1.061x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.926x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.032x ➖, 0↑ 0↓)
random-access / vortex-file-compressed / ns (0.958x ➖, 2↑ 0↓)
random-access / parquet / ns (1.003x ➖, 0↑ 0↓)
random-access / lance / ns (1.009x ➖, 0↑ 0↓)
|
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.059x ➖, 0↑ 3↓)
random-access / parquet / ns (1.090x ➖, 0↑ 3↓)
random-access / lance / ns (1.305x ❌, 0↑ 5↓)
|
Single-expression partitions are no longer wrapped in a one-field pack, and repeated sub-expressions within a partition share one output. Struct field projections therefore evaluate each field reader with its identity expression and recombine via the existing get_item/pack optimizer rules, replacing the reader-level FieldProjection fast path. The plan rewrites drop their collapse-single-field-pack post-pass, and single boolean partitions keep their Bool dtype. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FitS5Twto7znn1ek6JZKh6
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
Closing as superseded and stale. The PR targets the old Generated by Claude Code |
Summary
Plain multi-column struct projections rebuild partition packs and expression arrays for each batch. Recognize projections made entirely of root-field accesses, evaluate the child readers directly, and assemble the requested struct. Computed expressions retain the general evaluator.
Changes
Regression cases cover reordered and repeated fields, output nullability, parent nulls, row masks, and computed-projection fallback.
Validation
cargo test -p vortex-layout --lib --locked -j2 layouts::struct_::reader::— 12 passed.