Repository navigation
ci: upload sampled vx file trees from SQL benchmarks - #11444
Conversation
After each local SQL benchmark run, dump `vx inspect` and `vx tree layout --verbose` for a sample of the dataset's Vortex files and upload them as an artifact kept for 5 days. The sample, per Vortex format directory, is one file of every table (partitions such as lineitem_0001 share a table) plus a fixed number of further partitions drawn without replacement, weighted by file size. The draw is seeded with the commit so reruns inspect the same files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAYg7oWmiWHug452hr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAYg7oWmiWHug452hr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Cover the names every vortex-bench generator writes: split TPC-H, TPC-DS and nested TPC-H parts, the three ClickBench layouts, and the single-file datasets, including TPC-DS tables that share a prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAYg7oWmiWHug452hr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
777ee25 to
e903cc5
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.033x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.015x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.943x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.944x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 1↑ 1↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 1↑ 0↓)
|
Each SQL bench job uploads its own file-trees artifact. A final job now merges them into a single file-trees-<preset> zip, one directory per job, and deletes the per-job artifacts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAYg7oWmiWHug452hr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Dump every file of a format directory first, then keep one file of every table and add size-weighted partitions while their actual dumps fit in a line budget (--max-lines, default 20000), instead of drawing a fixed number of files. The manifest lists every file with its line count and whether its dump was kept. The dump step gets a 20-minute timeout so a large dataset cannot hold up the benchmark job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAYg7oWmiWHug452hr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Measured on the pr preset, a dumped tree line costs ~11 bytes zipped and a dataset's trees grow by 6-19 lines per MB of Vortex data. 200,000 lines (~2 MB zipped) keeps every file of the SF1 and SF10 datasets and samples about 89 of ClickBench's 100 files and the SF100 datasets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAYg7oWmiWHug452hr Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
develop (#11443, #11444) made "Capture file sizes" append file-size records to results.ingest.jsonl, which "Fetch baseline" reads, and added the sampled file tree dump and upload. The benchmark job now runs the capture alongside the dump, then the tree upload alongside the baseline fetch, so the fetch still sees the file-size records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
The SQL benchmarks write Vortex files that we never look inside. This PR adds one artifact per run containing
vxfile stats and layout trees for a sample of each dataset's files, so you can see which encodings and layouts a PR produced on real data.