Skip to content

Keep floating-point comparisons in DataFusion - #11379

Closed
danking wants to merge 1 commit into
developfrom
dk/dynamic-filter-signed-zero
Closed

danking wants to merge 1 commit into
developfrom
dk/dynamic-filter-signed-zero

Conversation

@danking

@danking danking commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Summary

DataFusion binary comparisons treat -0.0 and +0.0 as equal, while Vortex comparisons distinguish them. Pushing value = 0.0 into a Vortex scan can drop matching negative-zero rows; a residual filter cannot recover them. This also affects supported comparisons recovered from a live join filter beside an unsupported hash predicate.

Changes

Decline binary comparison conversion for floating operands, including dictionary value types, and leave exact evaluation in DataFusion. Floating arithmetic and IN conversion are retained; IN has distinct bitwise semantics and existing signed-zero/NaN coverage. The dynamic conjunct recovery and find_between optimization from #11215 are unchanged.

Add real Vortex DataSource regressions for static equality and a changing dynamic equality-plus-unsupported-modulo filter, both retaining a DataFusion residual and requiring both signed-zero rows.

Validation: pinned-nightly formatting and whitespace checks passed. The focused reproducer was exercised during the ColdFusion investigation; these two tests on the current upstream base have not been run locally and await CI. No throughput claim.

AI assistance: Codex investigated and prepared this patch.

Signed-off-by: Daniel King <dan@spiraldb.com>
@robert3005

Copy link
Copy Markdown
Contributor

can we add the datafusion float equals semantics? I think spark and paimon have the same

@danking

danking commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

You are probably a better Codex overseer for that than me. This was just a bug fix we did along the way to fast Viceberg.

are you imagining like a modes option on equality?

@robert3005

Copy link
Copy Markdown
Contributor

I need to decide if this is a mode or another expression implementation. If it's another expression implementation it's hard to mix them, if it's a mode it would be easier... let me prototype both

@robert3005

Copy link
Copy Markdown
Contributor

Full fix in #11401

@robert3005 robert3005 closed this Nov 11, 2026
@robert3005
robert3005 deleted the dk/dynamic-filter-signed-zero branch November 11, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants