Repository navigation
Store plan children in fixed-position slots - #11178
Conversation
Plan children now live in `PlanSlots`: fixed positions where an absent child is a `None` slot, as in array slots. Each slot is a `LazyPlanRef`, built on first access and shared by clones, which replaces the list-level `PlanChildren` initializer. - `Pack` slots are `[validity?, fields...]` and `ListPack` slots are `[elements, offsets, validity?]`, matching their layouts and arrays. A pruning `Zoned` plan keeps `[None, zones]`. - Lowering maps layout slots to plan slots in one helper, removing the per-layout index remapping. - `PlanRef::children()` yields the present slots in order. Compiling uses it, so pipeline inlets stay dense: inlet `i` reads the `i`-th present slot. `PackSource` now reads validity from its first inlet. - `with_children`/`child`/`child_name` become `with_slots`/`slot`/ `slot_name`. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCykJX86RM9UASFrjC Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`PlanSlots::present()` yields the present slots with their positions and is now the only place absent slots are dropped: `children()`, display, the optimizer's parent rules, the slot count Pack compiles, and Debug all go through it instead of filtering slots themselves. Eager slots are built by collecting `Option<PlanRef>`, so Pack and ListPack construct theirs in one expression. The optimizer maps every slot and keeps the result only when a child changed, replacing the copy-on-first-change loop. `reduce_parent` takes the child it rewrites instead of looking the slot up again, and PackSource takes validity off the front of its inlet arrays. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCykJX86RM9UASFrjC Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 14.07%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
`arrays.remove(0)` moved every field array down once per struct batch. Read inlet 0 straight into the validity and collect the fields from the remaining inlets instead. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCykJX86RM9UASFrjC Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 4, 2026
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.956x ➖, 2↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.977x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.048x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.015x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.024x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (1.021x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.010x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.996x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.013x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.043x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.027x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.824x ➖, 2↑ 0↓)
datafusion / vortex-compact / ns (0.857x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.755x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed / ns (0.925x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.953x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 1↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.012x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.995x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: JOB on NVME 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 1↑ 5↓)
duckdb / duckdb / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 1↓)
duckdb / duckdb / ns (0.996x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.998x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.995x ➖, 1↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.767x ➖, 9↑ 1↓)
datafusion / vortex-compact / ns (0.817x ➖, 4↑ 1↓)
datafusion / parquet / ns (0.822x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed / ns (0.847x ➖, 4↑ 0↓)
duckdb / vortex-compact / ns (0.807x ➖, 7↑ 0↓)
duckdb / parquet / ns (0.875x ➖, 3↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.521x ❌, 0↑ 2↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.044x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.872x ➖, 4↑ 1↓)
datafusion / vortex-compact / ns (0.907x ➖, 4↑ 1↓)
datafusion / parquet / ns (0.889x ➖, 3↑ 1↓)
duckdb / vortex-file-compressed / ns (0.822x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (0.876x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.916x ➖, 0↑ 1↓)
|
Plans with a fixed number of children built their slots from `vec![..]`, allocating a vector only to copy it into the inline slots. Build them from arrays instead, and reuse the vector's buffer when converting a `Vec<PlanRef>` whose slots spill. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCykJX86RM9UASFrjC Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Head branch was modified
Head branch was modified
Head branch was modified
Summary
Physical plan children used to be a packed list (
PlanChildren). An absent child, such as the validity of a non-nullable struct, shifted the positions of the children after it, so lowering had to remap layout slots to plan indices for each layout kind.Plans now store children in fixed-position slots, the same way arrays and layouts already do. An absent child is a
Noneslot, so a child's position never depends on which other children are present.Changes
Slots replace children (
slots.rs, replacingchildren.rs)PlanSlotsis aSmallVec<[Option<LazyPlanRef>; 4]>newtype.LazyPlanRefis oneArcholding aOnceCell<PlanRef>and an unsizedFninitializer.with_slotsandConcat::map_chunksnever lower a child twice.