Skip to content

Tuple: compact sketches are ordered by hash, with IS_ORDERED set, as in C++ - #812

Open
leerho wants to merge 2 commits into
mainfrom
tuple-compact-ordered
Open

leerho wants to merge 2 commits into
mainfrom
tuple-compact-ordered

Conversation

@leerho

@leerho leerho commented Nov 20, 2026

Copy link
Copy Markdown
Member

Makes the Java compact tuple sketch ordered by default, as C++ is, and writes the IS_ORDERED flag. As a result, Java and C++ compact tuple images are now byte-identical.

Problem

CompactTupleSketch.toByteArray() never set IS_ORDERED, and Java's compact tuple sketches kept hash-table order. C++ sorts by default and marks any sketch with 0 or 1 entries as ordered. So even an empty tuple image differed: flags were 0x0e in Java and 0x1e in C++.

Changes

  • Sorting: the CompactTupleSketch constructor sorts entries by hash, moving each summary with its hash. Every in-library result goes through it: compact(), TupleUnion, TupleIntersection, TupleAnotB and Filter.
    • The sort is a merge sort over the parallel arrays.
    • It returns at once if the entries are already sorted.
    • It never changes the arrays passed in; if they need sorting, it sorts copies. This matters because TupleAnotB.getResult() passes the operator's own hash array.
  • Tracking: CompactTupleSketch now records whether it is ordered, through a package-private isOrdered(). A deserialized sketch is ordered if its image has the flag or holds at most one entry. Otherwise it keeps the image's order, as with images from earlier Java releases.
  • Writing: toByteArray() sets IS_ORDERED when the sketch is ordered.
  • Javadoc: the set-operation and JaccardSimilarity javadoc said results are unordered; it now says they are ordered. The PreambleUtil format notes now describe the flag.
  • Compatibility: no Java reader reads IS_ORDERED, so this release and earlier ones accept the new images unchanged.

Not in this PR: the array-of-doubles tuple sketch, which has its own compact class and image layout. A strict empty-image check in the tuple reader, as theta has, will follow in a separate PR.

Testing

  • CompactTupleOrderedTest:
    • for n of 0, 1, 2, 17, 1,000 and 100,000, compact() is ordered, every summary stays with its hash, and the flags byte is exactly read-only | compact | ordered, plus empty when n = 0;
    • union, intersection and A-not-B results (stateless and stateful) are ordered, and a stateful A-not-B gives the same result twice;
    • the arrays passed to the constructor are unchanged;
    • an image with the ordered flag cleared reads back unordered and writes back unchanged, while 0- and 1-entry images read back ordered.
  • TupleCrossLanguageTest.checkCppBytesIdentical (group check_cpp_files): all eight tuple_int_n*_cpp.sk CL files (n = 0 to 1,000,000) are byte-identical to Java's images. On main, all eight differ at byte 5, the flags byte.
  • Full suite: 2,452 tests, 0 failures. The check_cpp_files profile passes too (17 tests). javac -Xlint:all -Xdoclint:all,-missing reports nothing in the tuple package.

🤖 Generated with Claude Code

- The CompactTupleSketch constructor used by compact() and by the union,
  intersection, A-not-B and filter results sorts the entries by hash,
  moving each summary with its hash. Arrays passed in are never changed:
  if they need sorting, sorted copies are used.
- CompactTupleSketch tracks whether it is ordered. A deserialized sketch
  is ordered if the image has the ordered flag, or holds at most one
  entry; otherwise it keeps the image's order.
- toByteArray() sets IS_ORDERED when the sketch is ordered.
- Javadoc that said results are unordered now says ordered.

Java images of compact tuple sketches now match C++ byte for byte; the
new check_cpp_files test compares all eight tuple_int CL files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Callers pass both arrays or neither; the check makes that explicit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants