Skip to content

Compare floats with the semantics of each query engine - #11401

Draft
robert3005 wants to merge 5 commits into
developfrom
rk/ieee754-compare
Draft

robert3005 wants to merge 5 commits into
developfrom
rk/ieee754-compare

Conversation

@robert3005

@robert3005 robert3005 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Builds on #11379. Instead of declining float pushdown, comparisons carry a CompareMode (SignedZeros × NanOrder × NestedNulls); the default stays the Vortex total order.

Engine Mode
DataFusion zeros equal, total order
DuckDB, Spark, Polars zeros equal, canonical NaN
PyArrow zeros equal, IEEE 754
Iceberg (Java API) zeros distinct, canonical NaN
  • Kernels and pruning apply only when the mode agrees with the total order.
  • Nested values with floats compare in the mode.
  • IN carries the mode too: both sides are normalized before the bitwise set lookup.
  • DuckDB nested comparisons order nested nulls last, as DuckDB does.
  • Polars predicates with NaN/inf literals are not pushed down (its JSON form loses them).

danking and others added 2 commits November 10, 2026 17:32
Signed-off-by: Daniel King <dan@spiraldb.com>
Add a CompareMode to comparisons with two settings: SignedZeros
(Distinct or Equal) and NanOrder (TotalOrder, Canonical or Ieee754).
The default is the Vortex total order. Encoding kernels and pruning
apply only when the mode gives the same result as the total order.

Pushdown uses the mode of each engine: DataFusion, DuckDB, Spark,
PyArrow and Polars. The Java API also has a mode for Iceberg.
Nested values that hold floats compare in the same mode.

Polars writes NaN and infinite float literals as null in its JSON
form. Restore them from the expression before pushdown.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 11, 2026
Polars writes these literals as null in its JSON form. Keep the old
fallback, which lets Polars evaluate the full predicate, until a better
conversion can read them.

Signed-off-by: Robert Kruszewski <github@robertk.io>
DuckDB orders a null inside a struct, list or map after all values.
Vortex orders it first. Add a NestedNulls setting (First or Last) to
CompareMode and use Last for nested DuckDB comparisons.

Kernels decline a comparison of nested values with nulls last, and
stats pruning does not rewrite it, because this order does not refine
the total order.

Signed-off-by: Robert Kruszewski <github@robertk.io>
* The native set compares the bits of floats, so a float {@code IN} becomes a disjunction of Spark equalities. A
* null element makes a non-match null, as in SQL.
*/
private static Expression floatIn(Expression field, DataType dataType, List<Expression> elements, boolean holdsNull) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should not do this

ListContainsOptions now holds a CompareMode. Before a bitwise membership
lookup, the elements and the needles are normalized so that two floats
have equal bits exactly when the mode makes them equal. With IEEE 754
semantics, a NaN needle gets other bits than a NaN element, so it
matches no element. The prepared set keeps one lookup set per
normalization.

DuckDB, Spark and Polars push float IN filters with the compare mode of
their engine, instead of a disjunction of equalities.

Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants