Repository navigation
Fast FastLanes Delta: fused decode, SIMD transposes, probe, filter, take and is_constant kernels - #11428
Conversation
Merging this PR will regress 1 benchmark2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
e121213 to
438088d
Compare
34f85ea to
1d8c9ac
Compare
This PR adds benchmarks for the Delta encoding. They are split out of #11428 so that its changes can be measured against `develop`, and #11428 stacks on this PR. ## Changes `encodings/fastlanes/benches/delta_decode.rs` adds 15 benchmarks. **Walltime (9 benchmarks).** `decode_bitpacked_deltas` decodes a 512 KiB Delta array whose deltas are bit-packed, for u16, u32 and u64. It carries `#[cpu_features]`, so it runs on each walltime CPU-feature leg, where the build selects the transpose and decode kernels. **Simulation (6 benchmarks).** These run once each, on 32 KiB arrays so they stay under 1 ms in CodSpeed's simulation, which runs roughly 13× slower than native: - `decode_primitive_deltas` decodes a u32 Delta array whose deltas are a primitive array. - `probe_once` and `probe_repeated` read 8 rows from 2 chunks of a 2048-row array. `probe_once` uses a one-off probe per row; `probe_repeated` reuses one probe for all 8. - `filter` selects one row in every 64, or one in every 4096. - `take` takes 128 scattered rows. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
1d8c9ac to
b369847
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.984x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.028x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.032x ➖, 1↑ 8↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.037x ➖, 0↑ 7↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
|
commented
Nov 13, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.070x ➖, 0↑ 9↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 1↓)
|
commented
Nov 13, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.983x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.972x ➖, 4↑ 3↓)
duckdb / parquet / ns (0.971x ➖, 0↑ 0↓)
|
commented
Nov 13, 2026
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.082x ➖, 1↑ 30↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.062x ➖, 0↑ 29↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
|
commented
Nov 13, 2026
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (1.065x ➖, 0↑ 3↓)
vortex / vortex-file-compressed / bytes (0.957x ➖, 4↑ 0↓)
vortex / parquet / ns (0.996x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.994x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
commented
Nov 13, 2026
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.028x ➖, 0↑ 1↓)
random-access / vortex-file-compressed / ns (1.014x ➖, 0↑ 2↓)
random-access / parquet / ns (0.991x ➖, 0↑ 0↓)
random-access / lance / ns (1.004x ➖, 0↑ 0↓)
random-access / litdata / ns (0.994x ➖, 0↑ 0↓)
|
commented
Nov 13, 2026
|
can you pull out the edition change. otherwise we are good to go |
commented
Nov 13, 2026
|
It's here 9e4c9ab |
Delta arrays whose deltas child is BitPacked (offset 0, no patches, global or blocked bit widths) now decode with fastlanes' `Delta::undelta_pack` kernel, which unpacks and prefix-sums each lane in one pass instead of first materializing the unpacked deltas. BitPacking is order-preserving (`unpack(pack(x)) == x`); its interleaved layout lives only inside the packed words. Delta's transpose is therefore the only transpose in the composition: the fused kernel yields the same transposed chunk as unpack followed by undelta, and each chunk is untransposed once, as before. The on-disk format is unchanged. Untransposing was ~85% of decode time, so both decode paths now use a gather-ordered untranspose that is ~5x faster than `fastlanes::Transpose::untranspose` and produces identical output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
The fused unpack and undelta decode now accepts any BitPacked deltas, or lazy slice of BitPacked deltas, whose first delta starts on a packed block. Delta slices its deltas at chunk boundaries, so this covers every slice Delta produces: whole packed blocks are decoded and the Delta offset is applied by slicing at the end. A patch changes every later value of its lane, so a chunk with patches is unpacked, patched and then prefix-summed; chunks without patches keep the fused kernel. Deltas that start inside a packed block still decode through the unfused path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
Delta compression and decompression pick fastlanes' transpose and untranspose variants once per array from the fearless_simd level: AVX-512 for u8 and u16, AVX2, SSE2, NEON, and the scalar loops otherwise. At the AVX-512 level u32 and u64 go through fastlanes' own dispatch, which keeps AVX2 on Zen 4 where AVX-512 is slower for those widths. This replaces the local gather-ordered untranspose. The fused decode now calls fastlanes' Delta::unchecked_undelta_pack in place of the local runtime bit-width dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
A Delta value depends on every earlier delta of its lane in its chunk,
so a read decodes the row's 1024-value chunk. A repeated probe keeps the
last decoded chunk, so reads within a chunk decode it once. Decoding a
chunk goes through the fused path when the deltas are bit-packed.
Reading the first 4096 rows of a u32 array over bit-packed deltas in
order (probe_once and probe_repeated in the delta_decode bench):
before after
one-off 7.6 ms 6.7 ms
repeated 7.9 ms 0.15 ms
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
Signed-off-by: Robert Kruszewski <github@robertk.io>
Delta decoding goes through a new DeltaChunks decoder, which prepares the
bases, the deltas and, for bit-packed deltas, the packed blocks and
per-chunk patches once, and then decodes any chunk on demand. Full
decoding writes every chunk straight into the output.
DeltaChunks implements ChunkDecoder, so the filter kernel reuses
filter_chunked: it decodes only the chunks that hold selected rows and
compacts each one while it is in cache. The take kernel groups the
valid indices by chunk with a counting sort and decodes each taken chunk
once; above one index per eight rows it decodes the whole array first,
as BitPacked does. Null indices read nothing, and out-of-bounds indices
are an error.
Filtering 512 KiB of u32 over bit-packed deltas (filter_decoded and
filter in the delta_decode bench):
selected rows decode, then filter filter kernel
1 in 64 27.9 us 19.5 us
1 in 4096 25.9 us 4.7 us
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
Signed-off-by: Robert Kruszewski <github@robertk.io>
An array with nulls is constant only if every value is null, which validity decides. Otherwise every value is its lane's base plus the earlier deltas of the lane, so zero deltas and equal bases make the array constant without decoding it. The converse holds only when the children hold exactly the visible values: a slice's first and last chunks can hold values outside it. So when the children are not constant, the kernel leaves the answer to the generic path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
Only the first and last chunks of a slice can hold values outside it. The chunks between them are wholly visible, so zero deltas and equal bases decide them exactly. The kernel now checks those chunks on the children, decodes just the visible rows of the partial first and last chunks, and compares them with the constant. A slice with no full chunk spans at most two partial chunks and is decoded. The kernel no longer leaves any all-valid array to the generic path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
Both encodings decode in 1024-value chunks through a ChunkDecoder, so their take now goes through one module, take_chunks. Positions that fall in one chunk are decoded together, through decode_indices when there are at most a sparse limit of them and otherwise by decoding the chunk: - take_in_order follows the positions as given. BitPacked uses it, as it did with its own loop, for indices and for the short runs of its range path; its PackedChunks decoder, so far used only by filter, now serves take too. - take_grouped groups the positions by chunk first, so each chunk is decoded once. Delta uses it, since reading any value of a chunk decodes all of it. Null indices read the array's first value instead of whatever their slot holds; the taken validity masks them. Delta checks that valid indices are in bounds. BitPacked still trusts its caller for that: a check costs as much as the take for small batches of u64 indices, which baseline x86-64 cannot compare in vectors. bitpacking_take is unchanged within noise, and Delta's take of 1024 scattered rows of 512 KiB of u32 takes 25 us. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD Signed-off-by: Robert Kruszewski <github@robertk.io>
9e4c9ab to
8136498
Compare
delta_compress filled nulls forward across chunk boundaries, so a chunk that starts with nulls began from the previous chunk's last value. For values that restart in every chunk, such as RLE indices, those leading slots became bases far above the next values, the residuals wrapped, and the compressor's span estimate rejected Delta. Nullable RLE indices then fell back to plain bit-packing or RunEnd. Fill a chunk's leading nulls with its first valid value instead. Their residuals are zero whether the values continue across chunks or restart. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
RLE indices count up within each 1024-value chunk, so Delta packs them into a few bits. With RunEnd allowed on them, the compressor picked RLE(indices = RunEnd(ends = Delta)), which every decode and filter has to expand twice. Enable the exclusion for the indices child. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
8136498 to
c005092
Compare
Summary
FastLanes Delta decoded slowly and had almost no compute kernels: every filter, take,
scalar_atoris_constantdecoded the whole array first. Vortex now has the APIs this needs (the probe API and execute kernels). This PR uses them, together with fastlanes 0.8.0's fused unpack-and-undelta and SIMD transpose kernels, so Delta is cheap enough to select by default.Changes
Decode
Delta(BitPacked)decodes each 1024-value chunk with fastlanes' fusedunchecked_undelta_pack. The deltas are no longer unpacked into a full intermediate array. This covers patched and block-width bit-packing.fearless_simd: AVX-512 for 8- and 16-bit values; for 32- and 64-bit values, fastlanes' own dispatch, because AVX-512 is slower there on Zen 4.DeltaChunksis a reusable chunk decoder with scratch space. The kernels below share it.Kernels
scalar_atdecodes only the chunk holding the row. A repeated probe caches the last decoded chunk.is_constantreads only the stored parts where it can: all-null, mixed-null, or all-zero interior deltas plus constant bases. Only the partial chunks at either end of a slice are decoded.take_chunks.rsis shared between BitPacked and Delta.Compression
delta_compressused to carry the previous chunk's last value into a chunk that starts with nulls. Values that restart in every chunk, such as RLE indices, then got wrapping residuals, and the compressor rejected Delta for them. Those leading nulls now take the chunk's first valid value.RLE(indices = RunEnd(ends = Delta)), which every decode and filter expands twice. This enables the exclusion that had been commented out.