Repository navigation
Conversation
An estimation-mode image stores n separately from the compactors, and deserialize() took it as given. Compaction keeps the total weight of the retained items, so n must equal it; a corrupt image could otherwise give a sketch whose n disagrees with its items. Both deserializers now check it, and also reject a total weight that overflows. Matches the REQ check in apache/datasketches-java#810. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage Report for CI Build 44804773861Coverage decreased (-0.003%) to 97.047%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
proost
reviewed
Nov 21, 2026
| for (const auto& compactor: compactors) { | ||
| const uint64_t num_items = compactor.get_num_items(); | ||
| const uint8_t lg_weight = compactor.get_lg_weight(); | ||
| if (num_items > 0 && (lg_weight >= 64 || num_items > ((std::numeric_limits<uint64_t>::max() - total_weight) >> lg_weight))) { |
Member
There was a problem hiding this comment.
shifting 64 or more is undefined behavior in C++. So num_items == 0 case can avoid this guard.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An estimation-mode REQ image stores n separately from the compactors, and
deserialize()took it as given. Compaction keeps the total weight of the retained items (restore_counts()relies on this), so n must equal the sum of each compactor's item count times its weight. A corrupt image could otherwise give a sketch whose n disagrees with its items, which skews every rank and quantile.Both deserializers (stream and bytes) now compute the total weight and throw
std::invalid_argumentif n differs from it, or if the total overflows 64 bits. Exact-mode images don't store n (it is the number of items), so nothing changes for them.Test, added to "deserialize rejects inconsistent compactors": n one above and one below the total weight is rejected from bytes and from a stream. It fails on master. All REQ tests pass under ASan and UBSan, including the existing valid-image cases with a million items in both rank accuracy modes.
The same check is in apache/datasketches-java#810. This PR is independent of #587 and merges cleanly with it and with #593 and #594.
🤖 Generated with Claude Code