Repository navigation
feat(file stats): Add new with_field_file_statistics to writer. - #10886
Conversation
83ca8b4 to
18bf5be
Compare
Merging this PR will improve performance by 22.34%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
|
Hi @thorfour -- thanks for the contribution! Can you rebase it since there have been filestats changes and then I can give it a look |
This allows overwriting the default or globally selected (via with_file_statistics) file level column statistics on a per-field basis. Allowing writers to record unique sets of statistics as they see fit. Signed-off-by: Thor <thor.hansen@dash0.com>
18bf5be to
7801294
Compare
This allows overwriting the default or globally selected (via with_file_statistics) file level column statistics on a per-field basis. Allowing writers to record unique sets of statistics as they see fit.
Summary
We're looking to record a custom bitmap file level statistic for a column to speed up file-level pruning based on that column selection. However adding this metric right now applies to all columns that share a dtype, and we don't desire to record bitmaps for other columns. We're looking for a way to tell our writers to record file level statistics only for certain columns to prevent wasted work and wasted space recording stats we wont utilize.
Changes
Adds a new
with_field_file_statisticsthat works as a column level override to default column stats or globally applied stats viawith_file_statistics.