Skip to content

perf(writer): a node's integer min/max reuse the stats the competition computed (#583) - #700

Merged
dfa1 merged 1 commit into
mainfrom
perf/583-reuse-minmax
Nov 8, 2026
Merged

dfa1 merged 1 commit into
mainfrom
perf/583-reuse-minmax

Conversation

@dfa1

@dfa1 dfa1 commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Part of #583 (does not close it): the second min/max pass.

CascadingCompressor.nodeStats scanned every cascaded integer array for its min and max, which the scheme competition had often just computed in ArrayAndStats (integer stats always carry them, from the same kernel). Rust reads them off the array's stats cache. The array's ArrayAndStats is now built once in encodeWithCtx and shared by the competition and the node stats; the bounds come from it when it already computed its stats, and from a scan when it did not (never forcing the run/distinct/frequency counts for them).

Output is byte-identical by construction (same kernel, same input); NodeStatsBoundsTest checks reused vs scanned bounds for every integer width, U64 past 2^63, nullable columns with nulls, an all-null and an empty one, and CascadingCompressorTest that a real encode's root bounds equal a scan's for nullable and unsigned columns. The node-stats comparisons against vortex-jni in integration pass unchanged.

Not measured: no benchmark figure, so no changelog entry.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV

…petition computed (#583)

CascadingCompressor.nodeStats scanned every cascaded integer array for its min and max, which
the competition had often just computed for ArrayAndStats (integer stats always carry them, from
the same kernel): Rust reads them off the array's stats cache. The array's ArrayAndStats is now
built once in encodeWithCtx and shared by the competition and the node stats; the bounds come from
it when it already computed its stats, and from a scan when it did not, never forcing the
run/distinct/frequency counts for them.

NodeStatsBoundsTest checks the reused bounds equal the scanned ones for every integer width, U64
past 2^63, nullable columns with nulls and an all-null one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 merged commit 0f63812 into main Nov 8, 2026
6 checks passed
@dfa1
dfa1 deleted the perf/583-reuse-minmax branch November 8, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant