Repository navigation
Implemented Inclusion of min/max for Floats, Doubles and Items Sorted Views. - #547
Conversation
Also fixed a problem I discovered in the deserialization for Boolean quantiles, but it would also have shown up with other types. The data was ok, but the max value would have been incorrect. Also fixed some problems with some of the tests. Along the way I also replaced "back-slash-n" with platform independent "LS" in the sketches I touched. Also moved some functions like getCDF(..) and getPMF(...) into parent interfaces as default functions. Now the three parent interfaces for Doubles SV, Floats SV and Items SV are all code-parallel.
|
Changes by file: /kll/ KllDirectCompactItemsSketch
KllDoublesSketch
KllFloatsSketch
KllHeapDoublesSketch
KllItemsSketch
KllMemoryValidate
/quantiles/ DoublesSketch
ItemsUtil
/quantilescommon/ DoublesSketchSortedView
FloatsSketchSortedView
GenericSortedView
IncludeMinMax
ItemsSketchSortedView
/req/ ReqSketch
/test/... /kll/ KllDirectCompactItemsSketch
KllMiscDoublesTest
KllMiscFloatsTest
KllMiscItemsTest
/quantilescommon/ IncludeMinMaxTest
|
|
jmalkin
left a comment
There was a problem hiding this comment.
A few minor things to clean up and extra asserts in new tests. All really small stuff.
Overall, I think this is a very useful change.
| final int offset = srcLevels[0]; | ||
| System.arraycopy(srcQuantiles, offset, quantiles, 0, numItems); | ||
| //Remove free space from both itemsArray and levels array | ||
| final int[] myLevelsArr = new int[srcLevelsArr.length]; |
There was a problem hiding this comment.
unclear why using this instead of srcNumLevels+1 as before, but gonna trust it's fine either way
There was a problem hiding this comment.
Without studying closely, the srcNumLevels + 1 makes it seem like we are increasing the size of the levelsArr, when we are not. It is the same size as the original, but with different offsets.
| this.totalN = sk.getN(); | ||
| } | ||
|
|
||
| //Used for testing |
There was a problem hiding this comment.
Constructors like this are ... not great. We should really be trying to find ways to test w/o special constructors like this. Not gonna "require changes" because of it, but just noting it.
There was a problem hiding this comment.
Some of the tests would be very difficult to reproduce without this.
| * @throws IllegalArgumentException if sketch is empty. | ||
| */ | ||
| double[] getPMF(T[] splitPoints, QuantileSearchCriteria searchCrit); | ||
| //double[] getPMF(T[] splitPoints, QuantileSearchCriteria searchCrit); |
There was a problem hiding this comment.
remove commented-out line?
| GenericSortedView.validateItems(splitPoints, getComparator()); | ||
| final double[] buckets = getCDF(splitPoints, searchCrit); | ||
| final int len = buckets.length; | ||
| for (int i = len; i-- > 1; ) { |
There was a problem hiding this comment.
can just use int i = buckets.length here. clearly not wrong, but not necessary to define len :)
| System.arraycopy(quantilesIn, 0, adjQuantiles, offset, lenIn); | ||
| System.arraycopy(cumWeightsIn,0, adjCumWeights, offset, lenIn); | ||
|
|
||
| //Adjust the low end if required. |
There was a problem hiding this comment.
asked on slack and we don't need to adjust the next item's weight cuz it's already cumulative here. but can we add a short comment about that for future reference?
| * @author Lee Rhodes | ||
| */ | ||
| @SuppressWarnings("unused") | ||
| //@SuppressWarnings("unused") |
There was a problem hiding this comment.
remove commented out line if not needed?
| } | ||
| final int topIn = quantiles.length - 1; | ||
| final int topAdj = adjQuantiles.length - 1; | ||
| assertEquals(adjQuantiles[topAdj], maxItem); |
There was a problem hiding this comment.
i think it'd be good to assert that the adjQuantiles array is 2 longer than the original quantiles array in here, too.
The printfs might be convient when debugging but they're basically no-ops in tests (and I treat them as such when reviewing). We want to make sure we assert on the expected
end-user behavior. And in this case, that's that the arrays are longer because we add in min and max.
…etches kll_sketch and quantiles_sketch keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, which compaction can drop the min and max from. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step: quantiles_sorted_view::include_min_max() adds a missing min or max as an entry of weight 1, taken from its neighbor, so the total weight does not change. kll_sketch and quantiles_sketch call it from get_sorted_view(). REQ does not change, because its get_rank() does not read the sorted view. The min and max tests build 1000 sketches and check the quantiles at ranks 0 and 1, also on a deserialized sketch and on its merge with a second sketch. Co-authored-by: Isaac <no-reply@databricks.com>
…EQ sketches The KLL, quantiles and REQ sketches keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step: the sketches call the new quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from get_sorted_view(). It appends a missing max with weight 1 taken from the entry before it, and adds a missing min with weight 1 taken from the entry after it, so the total weight does not change. The min is added in the same pass that makes the weights cumulative, so the entries shift without a second pass. REQ get_rank() summed the weights in the compactors and did not read the sorted view. Move it to the cached sorted view, as in KLL, quantiles and Java ReqSketch, and remove req_compactor::compute_weight(), which has no other caller. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. Keep get_sorted_view() as fast as before: the view reserves room for the min and max, but only for a non-empty sketch. add() returns early for an empty range (a KLL sketch usually has one or two empty levels once n is much larger than k) and uses emplace_back() instead of push_back() of a temporary. The new tests check the quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing. Co-authored-by: Isaac <no-reply@databricks.com>
…EQ sketches The KLL, quantiles and REQ sketches keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step: the sketches call the new quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from get_sorted_view(). It appends a missing max with weight 1 taken from the entry before it, and adds a missing min with weight 1 taken from the entry after it, so the total weight does not change. The min is added in the same pass that makes the weights cumulative, so the entries shift without a second pass. REQ get_rank() summed the weights in the compactors and did not read the sorted view. Move it to the cached sorted view, as in KLL, quantiles and Java ReqSketch, and remove req_compactor::compute_weight(), which has no other caller. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The view reserves room for the min and max, so adding them does not reallocate it. It reserves only for a non-empty sketch, so an empty sketch does not allocate. The new tests check the quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing. Co-authored-by: Isaac <no-reply@databricks.com>
…EQ sketches The KLL, quantiles and REQ sketches keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step: the sketches call the new quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from get_sorted_view(). It appends a missing max with weight 1 taken from the entry before it, and adds a missing min with weight 1 taken from the entry after it, so the total weight does not change. The min is added in the same pass that makes the weights cumulative, so the entries shift without a second pass. REQ get_rank() summed the weights in the compactors and did not read the sorted view. Move it to the cached sorted view, as in KLL, quantiles and Java ReqSketch, and remove req_compactor::compute_weight(), which has no other caller. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The view reserves room for the min and max, so adding them does not reallocate it. It reserves only for a non-empty sketch, so an empty sketch does not allocate. The new tests check the quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing. Co-authored-by: Isaac <no-reply@databricks.com>
…EQ sketches The KLL, quantiles and REQ sketches keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step: the sketches call the new quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from get_sorted_view(). It appends a missing max with weight 1 taken from the entry before it, and adds a missing min with weight 1 taken from the entry after it, so the total weight does not change. The min is added in the same pass that makes the weights cumulative, so the entries shift without a second pass. REQ get_rank() summed the weights in the compactors and did not read the sorted view. Move it to the cached sorted view, as in KLL, quantiles and Java ReqSketch, and remove req_compactor::compute_weight(), which has no other caller. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The view reserves room for the min and max, so adding them does not reallocate it. It reserves only for a non-empty sketch, so an empty sketch does not allocate. The new tests check the quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing. Co-authored-by: Isaac <no-reply@databricks.com>
…EQ sketches The KLL, quantiles and REQ sketches keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. REQ get_rank() summed the weights in the compactors and did not read the sorted view. Move it to the cached sorted view, as in KLL, quantiles and Java ReqSketch, and remove req_compactor::compute_weight(), which has no other caller. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
…EQ sketches The KLL, quantiles and REQ sketches keep the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. REQ get_rank() summed the weights in the compactors and did not read the sorted view. Move it to the cached sorted view, as in KLL, quantiles and Java ReqSketch, and remove req_compactor::compute_weight(), which has no other caller. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
kll_sketch keeps the exact min and max, but get_quantile(0) and get_quantile(1) read only the retained items, and compaction can drop the min and max from them. Since 6.0.0, datasketches-java adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547). Port that step for KLL. A missing min or max is added to the sorted view as an entry of weight 1, taken from its neighbor, so the total weight does not change. The levels are sorted, so get_sorted_view() finds out if the min is retained by comparing it with the first item of each level. If not, it adds the min with add_min() before the levels, so no entry has to move, and add() does not merge the first level into the lone min. convert_to_cummulative(max_item) takes the weight of the min from the next entry and appends a missing max. The view reserves room for both, but only for a non-empty sketch. The quantiles and REQ sketches do not change. For non-arithmetic types, the view and get_quantile() refer to the items of the sketch, now also to its min and max. Document that they must not be used after the sketch is updated, merged, assigned to, moved or destroyed. The new tests check the KLL quantiles at ranks 0 and 1 in estimation mode, for floats and strings, and after deserialize and merge, plus the view with the min, the max, both or neither missing, and with empty ranges. Co-authored-by: Isaac <no-reply@databricks.com>
Also fixed a problem I discovered in the deserialization for Boolean quantiles, but it would also have shown up with other types. The data was ok, but the max value would have been incorrect.
Also fixed some problems with some of the tests.
Along the way I also replaced "back-slash-n" with platform independent "LS" in the files I touched.
Also moved some functions like getCDF(..) and getPMF(...) into parent interfaces as default functions. Now the three parent interfaces for Doubles SV, Floats SV and Items SV are all code-parallel.