Repository navigation
Conversation
Compactor: - The copy and converting constructors release their storage and the copies made when a copy throws, or when the converting constructor rejects the order of the items. - merge() works out the sections and the capacity first and allocates once, then copies the other compactor's items and destroys the copies if one throws. Only then does it change the state and the sections. It used to change them first, and leaked the copies made before a copy that threw. It compares with the sketch's comparator, not C(). - compact() allocates the space in the next level, and the growth that more sections need, before it flips or draws the coin and changes anything. A failed compaction used to leave the coin flipped (or a random bit drawn), and could change the sections and then fail to grow. Sketch: - update() compares the item with min and max and copies it first, and sets min and max only after the item is in level zero. - merge() copies the new min and max before it sets either. The same gaps are addressed in datasketches-java by apache/datasketches-java#808, except the item-loss window in compress(), which C++ does not have: compact() reserved the next level's space before moving any item. Results are byte-identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage Report for CI Build 44790311547Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Coverage increased (+0.03%) to 97.018%Details
Uncovered ChangesNo uncovered changes found. Coverage Regressions2 previously-covered lines in 2 files lost coverage.
Coverage Stats
💛 - Coveralls |
A compactor that has never held an item has no storage, so serialize() passed a null pointer with a count of 0, and the default serde passed it on to memcpy(). UBSan on Linux reports that as undefined behavior. Serialize the items only when there are some; the bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if (sorted_ && !std::is_sorted(begin(), end(), comparator_)) { | ||
| destroy_items(); // the destructor doesn't run if a constructor throws | ||
| throw std::logic_error("items must be sorted"); | ||
| } |
There was a problem hiding this comment.
If comparator_ itself throws inside std::is_sorted, destroy_items() is not reached and the allocated buffer and copied items leak. Wrapping the post-copy_items check in try { ... } catch (...) { destroy_items(); throw; } avoids that.
There was a problem hiding this comment.
Fixed in a73df7c: the check is now wrapped, so the items are released if the comparator throws. Tested with a comparator that throws at each comparison in turn.
| sort(); | ||
| auto offset = hra_ ? capacity_ - num_items_ : num_items_; | ||
| auto from = hra_ ? begin() - other.get_num_items() : end(); | ||
| auto to = from + other.get_num_items(); | ||
| auto other_it = other.begin(); | ||
| for (auto it = from; it != to; ++it, ++other_it) new (it) T(conditional_forward<FwdC>(*other_it)); | ||
| if (!other.sorted_) std::sort(from, to, comparator_); | ||
| if (num_items_ > 0) std::inplace_merge(hra_ ? from : begin(), items_ + offset, hra_ ? end() : to, C()); | ||
| num_items_ += other.get_num_items(); | ||
| if (!other.sorted_) std::sort(from, from + num, comparator_); |
There was a problem hiding this comment.
Since other's items are now copied into [from, from + num) before sort(), self-merging (sketch.merge(sketch)) when level 0 is unsorted sets this->sorted_ = true (aliasing other.sorted_), skips std::sort(from, from + num, comparator_), and passes an unsorted range to std::inplace_merge (sketch.merge(std::move(sketch)) also moves out of *this before merging):
req_sketch<int> s(12, true);
s.update(1);
s.update(5);
s.merge(s); // leaves [1, 5, 1, 5] with sorted_ == trueThere was a problem hiding this comment.
Thanks, fixed in a73df7c: merging a sketch into itself (lvalue or rvalue) now merges a copy, which gives the same result as on master. Your example now gives 1 1 5 5. Classic quantiles has the same problem (on master too), and I will fix it here once #586 is merged, since it touches the same function.
| const uint32_t ne = nearest_even(ssr); | ||
| if (state_ >= static_cast<uint64_t>(1ULL << (num_sections_ - 1)) | ||
| && section_size_ > req_constants::MIN_K | ||
| if (state >= static_cast<uint64_t>(1ULL << (s.num - 1)) |
There was a problem hiding this comment.
For k = 1024, 4 doublings reach s.num = 48 (s.size = 256), which passes check_header (num_sections <= 64). If state >= (1ULL << 47) (for example on a deserialized sketch), next_sections doubles s.num to 96 (ne = 182 >= MIN_K), and the next iteration of while (next_sections(...)) in merge() evaluates 1ULL << 95 (UB, and serialize() writes num_sections = 96 which check_header rejects). Stopping once s.num > 32 keeps s.num <= 48 consistent with check_header.
There was a problem hiding this comment.
Fixed in a73df7c: the sections stop doubling at 48. This was on master as well (in ensure_enough_sections). The new test builds your case directly, a compactor with 48 sections of 256 and state 2^47, and fails under UBSan without the fix.
| write(os, padding); | ||
| write(os, num_items_); | ||
| serde.serialize(os, begin(), num_items_); | ||
| if (num_items_ > 0) serde.serialize(os, begin(), num_items_); // an empty compactor may have no storage |
There was a problem hiding this comment.
Symmetric case on the read side: deserialize_items (L518, L575) still calls alloc.allocate(0) and serde.deserialize(..., 0) when num == 0, leaving items_ != nullptr while capacity_ == 0 (and passing (nullptr, 0) to memcpy if an allocator returns nullptr for zero-sized allocations).
There was a problem hiding this comment.
Fixed in a73df7c: an empty compactor now gets no storage on deserialize, and the serde is not called for it. The test deserializes an empty compactor with an allocator that throws on any allocation.
| if (is_new_min) new_min = optional<T>(static_cast<const T&>(item)); | ||
| if (is_new_max) new_max = optional<T>(static_cast<const T&>(item)); | ||
| reset_sorted_view(); | ||
| compactors_[0].append(std::forward<FwdT>(item)); |
There was a problem hiding this comment.
If item aliases an element in this and compactors_[0] is at num_items_ == capacity_ (such as right after deserialize(), or after a failed compression filled the buffer), append() calls grow() and frees the backing array before constructing from item (heap-use-after-free under ASan, same pattern as #573):
req_sketch<std::string> s0(12);
s0.update(std::string(64, 'a'));
auto b = s0.serialize();
auto s = req_sketch<std::string>::deserialize(b.data(), b.size());
s.update(s.get_quantile(0.5)); // heap-use-after-free in append()Materializing T before grow() also avoids reallocating compactors_[0] when T's copy constructor throws.
There was a problem hiding this comment.
Fixed in a73df7c: when level zero has to grow, append() now copies the item first, then grows, then moves the copy in. A failed copy then also leaves level zero as it was. On macOS ASan does not see this (the std::string copy constructor is in the uninstrumented libc++ dylib), so the test checks the value instead: without the fix, the item added is the empty moved-from string.
| if (is_new_min) min_item_ = std::move(new_min); | ||
| if (is_new_max) max_item_ = std::move(new_max); | ||
| ++num_retained_; | ||
| ++n_; |
There was a problem hiding this comment.
Should update() and merge() also cap n_ at 2^63 - 1 as in #585? Without an overflow check, ~60 repeated merges wrap n_ to 0 (is_empty() becomes true while items are retained) and reach 1ULL << 64 UB at level 64.
| if (is_new_min) min_item_ = std::move(new_min); | ||
| if (is_new_max) max_item_ = std::move(new_max); |
There was a problem hiding this comment.
Committing min_item_ and max_item_ before merging any compactor leaves them modified even when compactors_[0].merge() throws on allocation or on the first item copy: an empty target stays is_empty() == true while holding live min_item_ / max_item_, and a 1-item target {10} failing to merge {1, 999} ends up with n == 1, sole item 10, min == 1, max == 999. In addition, if num_levels == 1 after remove_empty_levels_above(num_levels) in catch, exact-mode serialize() omits min_item_ and max_item_, so get_min_item() / get_max_item() disagree with a serialization round-trip.
There was a problem hiding this comment.
Fixed in a73df7c. merge() now merges the compactors from the highest level down (each compactor merge either completes or changes nothing), and sets min and max with the first items of other that the sketch takes. So if a merge fails, min and max are unchanged unless items of other were added; and if they were, a level above level zero holds items, so the sketch is in estimation mode and serializes min and max. The merge tests now check that min and max are unchanged when nothing was merged and that they survive a round trip. Serialized results are still byte-identical to master.
| // the converting constructor checks that the sorted levels are sorted by the comparator | ||
| const req_sketch<int, ordered_by> copy(sketch, ordered_by(true)); | ||
| REQUIRE(copy.get_n() == 2000); | ||
| REQUIRE(sketch.get_rank(1000) == Approx(0.5).margin(0.05)); |
There was a problem hiding this comment.
sketch.get_CDF(...) / get_PMF(...) still fail to compile with ordered_by (and use the wrong order for stateful default-constructible comparators) because quantiles_sorted_view::check_split_points (common/include/quantiles_sorted_view.hpp:204, 213) is static and constructs Comparator() instead of using comparator_.
There was a problem hiding this comment.
Fixed in a73df7c: check_split_points is now a member that uses the view's comparator. The ordered_by test now calls get_CDF() and get_PMF(), and checks that ascending split points are rejected under the descending order. This also fixes KLL and classic quantiles, which share the sorted view.
- merge(): merging a sketch into itself merges a copy, as merging read other while changing this sketch. - merge(): merge the compactors from the highest level down, and set min and max with the first items of other that the sketch takes. A failed merge then leaves min and max unchanged unless items of other were added, and in that case a level above level zero holds items, so the sketch stays in estimation mode, where min and max are serialized. - update() and merge(): cap n at req_constants::MAX_N = 2^63 - 1, as KLL does. - Compactor append(): when level zero must grow, copy the item first, since it may be one of the items that growing moves and frees. - Compactor: stop doubling the sections at 48, which keeps 1 << (num_sections - 1) defined and stays within what deserialize accepts. - Compactor converting constructor: release the items if the comparator throws while the order is checked. - Compactor deserialize: an empty compactor gets no storage. - quantiles_sorted_view: check split points with the view's comparator instead of a default-constructed one, so get_CDF() and get_PMF() compile with comparators that can't be default-constructed and use the right order with ones that have state. Serialized results are byte-identical to master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
merge() read other while it changed this sketch, so merging a sketch into itself used memory it had freed. It now merges a copy, as REQ does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This fixes the REQ gaps found while reviewing #582, and the C++ side of the gaps that apache/datasketches-java#808 addresses in Java.
Compactor:
merge()first works out the sections and capacity it will need, then allocates once. Next it copies the other compactor's items, destroying the copies if one throws. Only after that does it change the state and the sections. Before, it changed the state and sections first, and it leaked the copies made before a copy that threw. It also now compares with the sketch's comparator instead ofC().compact()now allocates the space in the next level, and any growth that more sections need, before it flips or draws the coin or changes anything. Before, a failed compaction left the coin flipped or a random bit drawn, and it could change the sections and then fail to grow. A failed compaction now leaves both compactors as they were and draws no random bit.Sketch:
update()compares the item with min and max and copies it first. It sets min and max only after the item is in level zero, so a failed append no longer leaves a min or max for an item that isn't retained.merge()copies the new min and max before it sets either.The docs of
update()andmerge()now say what a failure leaves behind. A comparator that throws while items are sorted or merged leaves the sketch undefined, as decided for Java in apache/datasketches-java#805.One difference from Java: C++ never had the window in which #808's
compress()could lose promoted items, becausecompact()already reserved space in the next level before moving any item.Tests:
const&merges: the target must stay consistent and usable;The five allocation tests fail on master and pass here. The comparator test needs this change to compile. Serialized results are byte-identical to master; I checked 600 seeded combinations of HRA and LRA, k = 4, 12 and 50,
const&and rvalue merges, and double and string sketches. All tests pass under ASan and UBSan.Added after review (a68f18e, a73df7c, 36789db):
merge()of a sketch into itself merges a copy. Classic quantiles does the same; on master, merging a classic quantiles sketch into itself used freed memory.merge()merges the compactors from the highest level down and sets min and max with the first items of other that it takes, so a failed merge leaves min and max unchanged unless items were added (and then the sketch is in estimation mode, where min and max are serialized).update()andmerge()cap n atreq_constants::MAX_N= 2^63 - 1, as KLL does since KLL: keep the sketch valid when an item copy throws; cap n at 2^63 - 1 #585.quantiles_sorted_view(common) checks split points with the view's comparator, soget_CDF()andget_PMF()work with comparators that can't be default-constructed, for KLL and classic quantiles too.Serialized results are still byte-identical to master.
🤖 Generated with Claude Code