Skip to content

Cross-check the child stats of string columns against vortex-jni #708

Open

Description

@dfa1

Split from #583. NodeStatsInteropIntegrationTest compares every flat root and the whole tree of the numeric columns against vortex-jni, but not the children of string columns (FSST lengths and codes, dict values).

Their stat values depend on the trained FSST symbol table, which cannot be byte-identical to Rust's: candidates of equal gain pop in hash-map iteration order, which depends on the CPU's SIMD width (see docs/compatibility.md, vortex.fsst). A plain comparison would be flaky.

To do: compare the stat set per node (which stats are present, precision) rather than values where the table matters, and values where they do not (e.g. vortex.sequence offsets, dict codes of a low-cardinality column).

Parity directive: this is a test gap, not a known divergence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions