Skip to content

REQ: reject n that isn't the total weight on deserialize - #595

Open
leerho wants to merge 1 commit into
masterfrom
req-deserialize-n
Open

leerho wants to merge 1 commit into
masterfrom
req-deserialize-n

Conversation

@leerho

@leerho leerho commented Nov 20, 2026

Copy link
Copy Markdown
Member

An estimation-mode REQ image stores n separately from the compactors, and deserialize() took it as given. Compaction keeps the total weight of the retained items (restore_counts() relies on this), so n must equal the sum of each compactor's item count times its weight. A corrupt image could otherwise give a sketch whose n disagrees with its items, which skews every rank and quantile.

Both deserializers (stream and bytes) now compute the total weight and throw std::invalid_argument if n differs from it, or if the total overflows 64 bits. Exact-mode images don't store n (it is the number of items), so nothing changes for them.

Test, added to "deserialize rejects inconsistent compactors": n one above and one below the total weight is rejected from bytes and from a stream. It fails on master. All REQ tests pass under ASan and UBSan, including the existing valid-image cases with a million items in both rank accuracy modes.

The same check is in apache/datasketches-java#810. This PR is independent of #587 and merges cleanly with it and with #593 and #594.

🤖 Generated with Claude Code

An estimation-mode image stores n separately from the compactors, and
deserialize() took it as given. Compaction keeps the total weight of the
retained items, so n must equal it; a corrupt image could otherwise give
a sketch whose n disagrees with its items. Both deserializers now check
it, and also reject a total weight that overflows. Matches the REQ check
in apache/datasketches-java#810.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 44804773861

Coverage decreased (-0.003%) to 97.047%

Details

  • Coverage decreased (-0.003%) from the base build.
  • Patch coverage: 1 uncovered change across 1 file (13 of 14 lines covered, 92.86%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
req/include/req_sketch_impl.hpp 14 13 92.86%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 17474
Covered Lines: 16958
Line Coverage: 97.05%
Coverage Strength: 1883065.86 hits per line

💛 - Coveralls

@leerho
leerho requested a review from proost November 20, 2026 21:07
for (const auto& compactor: compactors) {
const uint64_t num_items = compactor.get_num_items();
const uint8_t lg_weight = compactor.get_lg_weight();
if (num_items > 0 && (lg_weight >= 64 || num_items > ((std::numeric_limits<uint64_t>::max() - total_weight) >> lg_weight))) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shifting 64 or more is undefined behavior in C++. So num_items == 0 case can avoid this guard.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants