Repository navigation
Compare floats with the semantics of each query engine - #11401
Draft
robert3005 wants to merge 5 commits into
Draft
robert3005 wants to merge 5 commits into
robert3005 wants to merge 5 commits into
Conversation
Signed-off-by: Daniel King <dan@spiraldb.com>
Add a CompareMode to comparisons with two settings: SignedZeros (Distinct or Equal) and NanOrder (TotalOrder, Canonical or Ieee754). The default is the Vortex total order. Encoding kernels and pruning apply only when the mode gives the same result as the total order. Pushdown uses the mode of each engine: DataFusion, DuckDB, Spark, PyArrow and Polars. The Java API also has a mode for Iceberg. Nested values that hold floats compare in the same mode. Polars writes NaN and infinite float literals as null in its JSON form. Restore them from the expression before pushdown. Signed-off-by: Robert Kruszewski <github@robertk.io>
Polars writes these literals as null in its JSON form. Keep the old fallback, which lets Polars evaluate the full predicate, until a better conversion can read them. Signed-off-by: Robert Kruszewski <github@robertk.io>
DuckDB orders a null inside a struct, list or map after all values. Vortex orders it first. Add a NestedNulls setting (First or Last) to CompareMode and use Last for nested DuckDB comparisons. Kernels decline a comparison of nested values with nulls last, and stats pruning does not rewrite it, because this order does not refine the total order. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
commented
Nov 11, 2026
| * The native set compares the bits of floats, so a float {@code IN} becomes a disjunction of Spark equalities. A | ||
| * null element makes a non-match null, as in SQL. | ||
| */ | ||
| private static Expression floatIn(Expression field, DataType dataType, List<Expression> elements, boolean holdsNull) { |
Contributor
Author
There was a problem hiding this comment.
we should not do this
ListContainsOptions now holds a CompareMode. Before a bitwise membership lookup, the elements and the needles are normalized so that two floats have equal bits exactly when the mode makes them equal. With IEEE 754 semantics, a NaN needle gets other bits than a NaN element, so it matches no element. The prepared set keeps one lookup set per normalization. DuckDB, Spark and Polars push float IN filters with the compare mode of their engine, instead of a disjunction of equalities. Signed-off-by: Robert Kruszewski <github@robertk.io>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #11379. Instead of declining float pushdown, comparisons carry a
CompareMode(SignedZeros×NanOrder×NestedNulls); the default stays the Vortex total order.INcarries the mode too: both sides are normalized before the bitwise set lookup.