Skip to content

Add the nested TPC-H benchmark (tpch-nested) to vortex-bench - #11191

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
mprammer:mp/tpch-nested
Nov 6, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
mprammer:mp/tpch-nested

Conversation

@mprammer

@mprammer mprammer commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds tpch-nested, a port of Apache DataFusion Comet's nested TPC-H benchmark, to vx-bench. Each table keeps its join keys and one partition column (p_brand, c_mktsegment, o_orderdate, l_shipdate) at the top level and folds every other column into a one-element <table>_data: list<struct<...>>; each query unnests the lists it reads. Because every list holds exactly one element, the queries return the flat TPC-H answers and the suite reuses the flat row-count checks. It gives us a known external nested-data workload to measure the list layout against Parquet on both engines.

Changes

The data comes from the flat suite's generator with a nesting step (tpch::nest), so it keeps vx-bench's types, 1 Mi-row row groups, and file splits rather than Comet's Spark output; nested_schema reproduces Comet's per-table column placement and is unit-tested against it. The 22 queries in sql/tpch-nested/list/ are the flat vx-bench queries with Comet's derived-table rewrite (unnest(lineitem_data) as lineitem_data, then lineitem_data.l_quantity), one SQL text per query shared by DataFusion and DuckDB. The suite takes a shape option (only list today) so a differently nested variant of the same data can be added without renaming anything. The list layout needs the unstable bench edition and the V1 scan, so the orchestrator pins this suite to --scan-v1 true like the other list-layout suites. CI runs SF1 on NVMe in pr-full and develop and SF100 on NVMe and S3 in nightly, mirroring flat TPC-H; every lane in those presets was run at SF1 locally with row counts validated.

The first commit is the DuckDB exporter fix from #11190, which this suite needs for the DuckDB lanes; it drops out on rebase once that merges.

🤖 Generated with Claude Code

@mprammer
mprammer requested review from mhk197 and robert3005 November 4, 2026 04:27
@mprammer
mprammer marked this pull request as ready for review November 4, 2026 16:30
@robert3005

Copy link
Copy Markdown
Contributor

can you rebase on develop now? I have fixed the export bug

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Martin Prammer <martin@spiraldb.com>
@mprammer mprammer added the changelog/chore A trivial change label Nov 6, 2026

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to wire that into the runner but this gives us the scaffolding for the benchmark itself

@robert3005

Copy link
Copy Markdown
Contributor

What I meant is we need to add it to github actions orchestration

@robert3005
robert3005 merged commit 0e8eefe into vortex-data:develop Nov 6, 2026
100 of 102 checks passed
lwwmanning added a commit that referenced this pull request Nov 21, 2026
…ed TPC-H S3 data path (#11726)

This changes the nightly SQL benchmark matrix in
`bench-orchestrator/bench_orchestrator/ci_matrix/catalog.py`. The
develop and PR presets are unchanged.

Before, nightly ran DataFusion's datasource integration on the v1 scan
for every benchmark, which #11685 added. On its first night, six of
those jobs failed. TPC-DS SF=10 hung on q62 until the 120-minute job
timeout, and the runner shut down during TPC-DS SF=100 q62 and TPC-H
SF=100 q18. After, nightly skips datasource v1. The file v1 scans on
DataFusion and DuckDB stay, because they pass. Develop and `pr-scan`
still run datasource v1 on ClickBench and TPC-H SF=1.

Before, Nested TPC-H on S3 failed every night since #11191. prepare-data
wrote the data under `tpch-nested/100/list`, but the upload read
`tpch-nested/100.0/list`. After, the upload reads the directory that
prepare-data writes. A new test checks that each S3 data path names the
scale factor its job passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Will Manning <will@willmanning.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants