Skip to content

feat(file stats): Add new with_field_file_statistics to writer. - #10886

Merged
joseph-isaacs merged 1 commit into
developfrom
file-stats-column-selection
Nov 19, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
file-stats-column-selection

Conversation

@thorfour

Copy link
Copy Markdown
Contributor

This allows overwriting the default or globally selected (via with_file_statistics) file level column statistics on a per-field basis. Allowing writers to record unique sets of statistics as they see fit.

Summary

  • Tracking Issue: #000

We're looking to record a custom bitmap file level statistic for a column to speed up file-level pruning based on that column selection. However adding this metric right now applies to all columns that share a dtype, and we don't desire to record bitmaps for other columns. We're looking for a way to tell our writers to record file level statistics only for certain columns to prevent wasted work and wasted space recording stats we wont utilize.

Changes

Adds a new with_field_file_statistics that works as a column level override to default column stats or globally applied stats via with_file_statistics.

@thorfour thorfour added the changelog/feature A new feature label Oct 27, 2026
@thorfour
thorfour force-pushed the file-stats-column-selection branch from 83ca8b4 to 18bf5be Compare October 27, 2026 17:13
@codspeed

codspeed Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 22.34%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
✅ 1913 untouched benchmarks
⏩ 47 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation missing_stats[16] 92.6 µs 75.7 µs +22.34%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing file-stats-column-selection (7801294) with develop (114500b)

Open in CodSpeed

Footnotes

  1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@mhk197

mhk197 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Hi @thorfour -- thanks for the contribution! Can you rebase it since there have been filestats changes and then I can give it a look

This allows overwriting the default or globally selected (via
with_file_statistics) file level column statistics on a per-field basis.
Allowing writers to record unique sets of statistics as they see fit.

Signed-off-by: Thor <thor.hansen@dash0.com>
@thorfour
thorfour force-pushed the file-stats-column-selection branch from 18bf5be to 7801294 Compare November 5, 2026 19:29
@thorfour
thorfour requested a review from mhk197 November 5, 2026 20:41
@thorfour

thorfour commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Hi @thorfour -- thanks for the contribution! Can you rebase it since there have been filestats changes and then I can give it a look

@mhk197 done!

@joseph-isaacs
joseph-isaacs merged commit d2bcc26 into develop Nov 19, 2026
73 of 74 checks passed
@joseph-isaacs
joseph-isaacs deleted the file-stats-column-selection branch November 19, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants