diff --git a/CLAUDE.md b/CLAUDE.md index a4dac507..a94dfe22 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -83,7 +83,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy. - **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place. - **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide. - **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card. -- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. +- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. In a query summary, the penalty is not enough: a series that misses a query ranks below every series that ran them all, so a series cannot rank first by failing or skipping the hardest query (Lance skips TPC-DS q72). - **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline. - **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base. diff --git a/docs/bench/CONTRACT.md b/docs/bench/CONTRACT.md index e27c06ee..adcc64fe 100644 --- a/docs/bench/CONTRACT.md +++ b/docs/bench/CONTRACT.md @@ -64,6 +64,11 @@ column-by-column; an unknown `kind` or a missing dimension field fails the inges `compression_size` records include `uncompressed_bytes`, the Arrow memory size after the source Parquet file is decoded. The column is nullable so historical rows remain valid. +`compress-bench` also writes two size-only formats, with no `compression_time` records: +`parquet-snappy` (Parquet with Snappy, and otherwise the same writer settings as `parquet`, +which is zstd level 3) and +`vortex-compact` (Vortex with the compact strategy). Each is one write per dataset that is not +timed. `string-bench` shares the two compression tables with `compress-bench`. Its records use the dataset `strings/`, the string encoder (`onpair-12`, `fsst`, ...) as `dataset_variant`, @@ -73,11 +78,29 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t `fsst` label are part of the contract. `file_size` records are not written by `vortex-bench`: the monorepo's -`scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with +`scripts/capture-file-sizes.py` writes one per data file a SQL benchmark leaves on disk, with `benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format -directory) and `value_bytes`. The read service charts each benchmark and scale factor in the -`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop -baseline for the PR comment's file-size section. +directory) and `value_bytes`. A file's `format` is the name of its format directory, not its +codec. `parquet` is the suite's base Parquet: zstd level 3 where the benchmarks write it, and the +dataset's published files where the suite prescribes them, as ClickBench's are Snappy. CI +reads the records back as the develop baseline for the PR comment's file-size section. + +The read service charts both tables in the `Compression & File Size` group, one line per +format. It charts every current `compress-bench` dataset in `compression_sizes`, and three +`file_sizes` scopes, each summed over its files: ClickBench, TPC-H at SF10, and TPC-DS at SF1 +(`src/bench/lib/file-sizes.ts`). Its summary compares each format with zstd Parquet on the +`compress-bench` datasets and on TPC-DS. It leaves out `duckdb`, DuckDB's database file, which +the group still charts. On develop, TPC-DS SF1 records the sizes of Lance, Arrow IPC, and Snappy +Parquet (`parquet-snappy`) beside Parquet and Vortex. It queries Lance on DataFusion, and writes +Arrow IPC and Snappy Parquet after its queries, for their sizes only. Lance skips q72, which it cannot run; the query summary ranks a series +that misses a query below every series that ran them all. + +The `SQL Benchmark File Sizes` group charts every `file_sizes` scope, at every scale factor, and +leaves out `compression_sizes`. Its summary compares each format with Parquet, at each format's +newest commit that also has Parquet, on the benchmarks that write Parquet with bench's settings: +JOB, Polar Signals, statpopgen, TPC-DS, TPC-H, and nested TPC-H +(`FILE_SIZE_RANKED_BENCHMARKS`). It counts each benchmark once, as the geometric mean of its +scale factors, and shows no totals. `random_access_time` records include `open_mode`. The value is `cached` for a reused accessor or `reopen` when each timed take includes a new accessor. Historical rows use `cached`. diff --git a/docs/bench/architecture/read-path.md b/docs/bench/architecture/read-path.md index c7c1afac..32a83b5e 100644 --- a/docs/bench/architecture/read-path.md +++ b/docs/bench/architecture/read-path.md @@ -120,14 +120,15 @@ the v3 Axum server (so the frontend is generation-agnostic). The landing-page order is the curated `GROUP_ORDER` in `queries.ts`: ``` -Compression, Compression Size, Random Access, Clickbench, +Compression, Compression & File Size, Random Access, Clickbench, TPC-H (NVMe/S3) SF=1, SF=10, SF=100 (alternating), TPC-DS (NVMe) SF=1, Statistical and Population Genetics, Polar Signals Profiling, fineweb (NVMe/S3), Appian (NVMe) ``` -Groups not in the list sort last, alphabetically. +Groups not in the list sort after it, alphabetically, and then come the groups in +`GROUP_ORDER_LAST`: SQL Benchmark File Sizes, String Encoding, and String Encoding Size. ### Database connection diff --git a/src/app/(bench)/bench/page.tsx b/src/app/(bench)/bench/page.tsx index 8dc0adb3..36a2b097 100644 --- a/src/app/(bench)/bench/page.tsx +++ b/src/app/(bench)/bench/page.tsx @@ -134,6 +134,7 @@ function HomeContent({ data: [params, groups, universe] }: { data: LandingData } /> ({ name: group.name, slug: group.slug }))} />
+
{groups.length === 0 ? (

No data ingested yet.

) : ( @@ -181,7 +182,6 @@ function HomeContent({ data: [params, groups, universe] }: { data: LandingData } )}
-