Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ce0e90d
Merge the size groups into one File Sizes group on bench
lwwmanning Nov 20, 2026
17784d1
Keep the SQL File Sizes group beside File Sizes on bench
lwwmanning Nov 20, 2026
cca5c97
Note the TPC-DS size-only formats in the bench contract
lwwmanning Nov 20, 2026
8d90c34
Rank a query series that skips a query below complete ones on bench
lwwmanning Nov 20, 2026
b9b9a1f
Block Vercel's preview Toolbar in Lighthouse runs
lwwmanning Nov 20, 2026
27097f8
Explain bench's two Parquet variants on the methodology page
lwwmanning Nov 21, 2026
86a487c
Name DuckDB 2.0 as the writer of Parquet (Snappy) on bench
lwwmanning Nov 21, 2026
c0cccc5
Explain how bench's Parquet is tuned on the methodology page
lwwmanning Nov 21, 2026
6ab4667
Merge remote-tracking branch 'origin/main' into wm/consolidate-file-s…
lwwmanning Nov 21, 2026
fc1dc1c
Measure a size family only when a format has all of it, and bump the …
lwwmanning Nov 21, 2026
f5eb6db
Rename File Sizes to File Sizes & Compression Ratios, and leave DuckD…
lwwmanning Nov 21, 2026
3021f81
Rename the size group to Compression & File Size
lwwmanning Nov 21, 2026
49ac480
Keep the renamed size groups' anchors, and describe the size ranking
lwwmanning Nov 21, 2026
bc7eae7
Label zstd Parquet as parquet-zstd in the Compression & File Size group
lwwmanning Nov 21, 2026
072de3e
Order the size summary by its Parquet score, and document its Arrow r…
lwwmanning Nov 21, 2026
f7d36b5
Move the methodology link and build commit above the first group
lwwmanning Nov 21, 2026
a4c669c
Name ClickBench on the group's renamed parquet chip
lwwmanning Nov 21, 2026
ec85c29
Rank SQL Benchmark File Sizes against parquet, one score per benchmark
lwwmanning Nov 21, 2026
63068ac
Show a partial file size score as not measured on the homepage
lwwmanning Nov 21, 2026
2e0d895
Count a SQL benchmark as measured when a format has any of its scale …
lwwmanning Nov 21, 2026
c0c959e
Rank DuckDB's file in the SQL size summary when it has every benchmark
lwwmanning Nov 21, 2026
7711640
Mark a size headline partial when Vortex lacks some datasets
lwwmanning Nov 21, 2026
9fceee3
Show Parquet's codecs as parquet (zstd) and parquet (snappy)
lwwmanning Nov 21, 2026
65f672f
Name zstd Parquet in size headlines, and weigh llms.txt's SQL sizes b…
lwwmanning Nov 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy.
- **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place.
- **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide.
- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card.
- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals.
- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. In a query summary, the penalty is not enough: a series that misses a query ranks below every series that ran them all, so a series cannot rank first by failing or skipping the hardest query (Lance skips TPC-DS q72).
- **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline.
- **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base.

Expand Down
31 changes: 27 additions & 4 deletions docs/bench/CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ column-by-column; an unknown `kind` or a missing dimension field fails the inges

`compression_size` records include `uncompressed_bytes`, the Arrow memory size after the
source Parquet file is decoded. The column is nullable so historical rows remain valid.
`compress-bench` also writes two size-only formats, with no `compression_time` records:
`parquet-snappy` (Parquet with Snappy, and otherwise the same writer settings as `parquet`,
which is zstd level 3) and
`vortex-compact` (Vortex with the compact strategy). Each is one write per dataset that is not
timed.

`string-bench` shares the two compression tables with `compress-bench`. Its records use the
dataset `strings/<column>`, the string encoder (`onpair-12`, `fsst`, ...) as `dataset_variant`,
Expand All @@ -73,11 +78,29 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t
`fsst` label are part of the contract.

`file_size` records are not written by `vortex-bench`: the monorepo's
`scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with
`scripts/capture-file-sizes.py` writes one per data file a SQL benchmark leaves on disk, with
`benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format
directory) and `value_bytes`. The read service charts each benchmark and scale factor in the
`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop
baseline for the PR comment's file-size section.
directory) and `value_bytes`. A file's `format` is the name of its format directory, not its
codec. `parquet` is the suite's base Parquet: zstd level 3 where the benchmarks write it, and the
dataset's published files where the suite prescribes them, as ClickBench's are Snappy. CI
reads the records back as the develop baseline for the PR comment's file-size section.

The read service charts both tables in the `Compression & File Size` group, one line per
format. It charts every current `compress-bench` dataset in `compression_sizes`, and three
`file_sizes` scopes, each summed over its files: ClickBench, TPC-H at SF10, and TPC-DS at SF1
(`src/bench/lib/file-sizes.ts`). Its summary compares each format with zstd Parquet on the
`compress-bench` datasets and on TPC-DS. It leaves out `duckdb`, DuckDB's database file, which
the group still charts. On develop, TPC-DS SF1 records the sizes of Lance, Arrow IPC, and Snappy
Parquet (`parquet-snappy`) beside Parquet and Vortex. It queries Lance on DataFusion, and writes
Arrow IPC and Snappy Parquet after its queries, for their sizes only. Lance skips q72, which it cannot run; the query summary ranks a series
that misses a query below every series that ran them all.

The `SQL Benchmark File Sizes` group charts every `file_sizes` scope, at every scale factor, and
leaves out `compression_sizes`. Its summary compares each format with Parquet, at each format's
newest commit that also has Parquet, on the benchmarks that write Parquet with bench's settings:
JOB, Polar Signals, statpopgen, TPC-DS, TPC-H, and nested TPC-H
(`FILE_SIZE_RANKED_BENCHMARKS`). It counts each benchmark once, as the geometric mean of its
scale factors, and shows no totals.

`random_access_time` records include `open_mode`. The value is `cached` for a reused accessor
or `reopen` when each timed take includes a new accessor. Historical rows use `cached`.
Expand Down
5 changes: 3 additions & 2 deletions docs/bench/architecture/read-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,15 @@ the v3 Axum server (so the frontend is generation-agnostic). The landing-page
order is the curated `GROUP_ORDER` in `queries.ts`:

```
Compression, Compression Size, Random Access, Clickbench,
Compression, Compression & File Size, Random Access, Clickbench,
TPC-H (NVMe/S3) SF=1, SF=10, SF=100 (alternating),
TPC-DS (NVMe) SF=1,
Statistical and Population Genetics, Polar Signals Profiling,
fineweb (NVMe/S3), Appian (NVMe)
```

Groups not in the list sort last, alphabetically.
Groups not in the list sort after it, alphabetically, and then come the groups in
`GROUP_ORDER_LAST`: SQL Benchmark File Sizes, String Encoding, and String Encoding Size.

### Database connection

Expand Down
2 changes: 1 addition & 1 deletion src/app/(bench)/bench/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ function HomeContent({ data: [params, groups, universe] }: { data: LandingData }
/>
<GroupNav groups={groups.map((group) => ({ name: group.name, slug: group.slug }))} />
<main id="main" tabIndex={-1}>
<Footer lead />
{groups.length === 0 ? (
<p className="empty">No data ingested yet.</p>
) : (
Expand Down Expand Up @@ -181,7 +182,6 @@ function HomeContent({ data: [params, groups, universe] }: { data: LandingData }
)}
</main>
<LandingState />
<Footer />
</>
);
}
13 changes: 12 additions & 1 deletion src/bench/components/Chart.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
chartAriaLabel,
chartPriority,
LAZY_HYDRATION_ROOT_MARGIN,
type SeriesLabels,
throttle,
ZOOM_THROTTLE_MS,
} from '@/bench/lib/chart-format';
Expand Down Expand Up @@ -75,13 +76,22 @@ export interface ChartIslandProps {
groupSlug?: string;
/** Server-fetched payload on the permalink page (skips the initial fetch). */
initialPayload?: ChartResponse;
/** Display names the chart gives some of its formats (see `SeriesLabels`). */
seriesLabels?: SeriesLabels | undefined;
Comment thread
lwwmanning marked this conversation as resolved.
Resolved
}

/**
* One interactive chart card. Renders the full card chrome (the v3
* `chart_card` markup) and drives Chart.js through a [`ChartController`].
*/
export function Chart({ slug, name, index, groupSlug, initialPayload }: ChartIslandProps) {
export function Chart({
slug,
name,
index,
groupSlug,
initialPayload,
seriesLabels,
}: ChartIslandProps) {
const base = useBenchBase();
const cardRef = useRef<HTMLElement>(null);
const canvasRef = useRef<HTMLCanvasElement>(null);
Expand Down Expand Up @@ -192,6 +202,7 @@ export function Chart({ slug, name, index, groupSlug, initialPayload }: ChartIsl
stripWindow: stripWindowRef.current,
}),
{ setY, setLoading, setError, setRetryable, setConstructed, setLatestValues },
seriesLabels,
);
controllerRef.current = controller;
if (initialPayload) {
Expand Down
7 changes: 5 additions & 2 deletions src/bench/components/Footer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,18 @@ import { MethodologyLink } from '@/bench/components/MethodologyLink';
* with the deployed commit. When the SHA is the literal `"unknown"` (a build
* outside a git checkout, e.g. local dev), the short SHA renders without a
* commit link, matching v3. It opens with the link to the methodology page.
*
* With `lead`, the same line opens the landing page's `main`, above the first
* group, where readers find it without scrolling past every group.
*/
export function Footer() {
export function Footer({ lead = false }: { lead?: boolean }) {
const fullSha = process.env.VERCEL_GIT_COMMIT_SHA ?? 'unknown';
const shortSha = fullSha.slice(0, 7);
const commitUrl = `https://github.com/vortex-data/vortex.dev/commit/${fullSha}`;
return (
// A div, not a footer: the system's Footer below it is the page's one
// contentinfo landmark.
<div className="site-footer">
<div className={lead ? 'site-footer site-footer--lead' : 'site-footer'}>
<MethodologyLink />
<span aria-hidden="true"> · </span>
<span className="site-footer-label">build </span>
Expand Down
20 changes: 18 additions & 2 deletions src/bench/components/GroupBody.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { useEffect, useLayoutEffect, useRef, useState } from 'react';
import { useBenchBase } from '@/bench/components/BenchBase';
import { Chart } from '@/bench/components/Chart';
import { GroupToolbar } from '@/bench/components/GroupToolbar';
import { chartPriority, type FilterUniverse } from '@/bench/lib/chart-format';
import { chartPriority, type FilterUniverse, type SeriesLabels } from '@/bench/lib/chart-format';
import { abortGroupBundle, ensureGroupBundle } from '@/bench/lib/chart-store';
import {
noteGroupBodyClosed,
Expand All @@ -30,6 +30,8 @@ export interface GroupBodyProps {
universe: FilterUniverse;
/** The filter choices the page URL gives this group (`?group=…&hide=…`). */
initialFilter: GroupFilterChoices;
/** Display names the toolbar gives some formats (see `SeriesLabels`). */
seriesLabels?: SeriesLabels | undefined;
}

/**
Expand Down Expand Up @@ -58,6 +60,7 @@ export function GroupBody({
startIndex,
universe,
initialFilter,
seriesLabels,
}: GroupBodyProps) {
const base = useBenchBase();
const ref = useRef<HTMLDivElement>(null);
Expand Down Expand Up @@ -120,7 +123,19 @@ export function GroupBody({
<div className="group-body" ref={ref}>
{open && (
<>
<GroupToolbar groupSlug={groupSlug} universe={universe} />
<GroupToolbar
groupSlug={groupSlug}
universe={universe}
seriesLabels={seriesLabels}
unlabeledCharts={Object.fromEntries(
Object.keys(seriesLabels ?? {}).map((format) => [
format,
charts
.filter((link) => link.seriesLabels?.[format] === undefined)
.map((link) => link.name),
]),
)}
/>
<div className="chart-grid">
{charts.map((link, i) => (
<Chart
Expand All @@ -129,6 +144,7 @@ export function GroupBody({
name={link.name}
index={startIndex + i}
groupSlug={groupSlug}
seriesLabels={link.seriesLabels}
/>
))}
</div>
Expand Down
5 changes: 3 additions & 2 deletions src/bench/components/GroupSection.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ export function GroupSection({
// selected by the dataset name inside the group's slug (`null` if the slug
// does not parse or no matching explainer exists).
const docUrl = groupDocUrl(group.slug);
const headline = groupHeadline(group.summary);
const headline = groupHeadline(group.summary, group.seriesLabels);
return (
<section
className="group-details"
Expand Down Expand Up @@ -130,13 +130,14 @@ export function GroupSection({
</span>
</summary>
</details>
<SummaryCard summary={group.summary} />
<SummaryCard summary={group.summary} seriesLabels={group.seriesLabels} />
<GroupBody
groupSlug={group.slug}
charts={group.charts}
startIndex={startIndex}
universe={universe}
initialFilter={initialFilter}
seriesLabels={group.seriesLabels}
/>
</section>
);
Expand Down
18 changes: 18 additions & 0 deletions src/bench/components/GroupToolbar.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,24 @@ describe('GroupToolbar (server-rendered markup)', () => {
});
});

describe('GroupToolbar renamed chips', () => {
it('names the charts where a renamed chip keeps the real name', () => {
const html = renderToStaticMarkup(
<GroupToolbar
groupSlug="toolbar-renamed"
universe={UNIVERSE}
seriesLabels={{ parquet: 'parquet (zstd)', vortex: 'vortex (default)' }}
unlabeledCharts={{ parquet: ['clickbench'], vortex: [] }}
/>,
);
expect(html).toContain('>parquet (zstd)<');
expect(html).toContain('title="parquet (zstd), and parquet on clickbench"');
// A renamed chip that every chart renames has no title.
expect(html).toContain('>vortex (default)<');
expect(html).not.toMatch(/data-value="vortex"[^>]*title=/);
});
});

describe('GroupToolbar empty-charts toggle', () => {
// The module-scope group store outlives renders, so noting empty charts
// before rendering mirrors a bundle fetch having classified the group.
Expand Down
66 changes: 63 additions & 3 deletions src/bench/components/GroupToolbar.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import { useCallback, useEffect, useId, useRef, useState, useSyncExternalStore } from 'react';

import { FilterIcon } from '@/bench/components/icons';
import { DEFAULT_VISIBLE, type FilterUniverse } from '@/bench/lib/chart-format';
import { DEFAULT_VISIBLE, type FilterUniverse, type SeriesLabels } from '@/bench/lib/chart-format';
import {
applyGroupMacro,
clearGroupFilterRow,
Expand Down Expand Up @@ -51,9 +51,18 @@ import { returnFocusToTrigger } from '@/bench/lib/panel-focus';
export function GroupToolbar({
groupSlug,
universe,
seriesLabels,
unlabeledCharts = {},
}: {
groupSlug: string;
universe: FilterUniverse;
/** Display names the chips give some formats (see `SeriesLabels`). */
seriesLabels?: SeriesLabels | undefined;
/**
* For each format the group renames, its charts that keep the real name, which the renamed
* chip's title lists.
*/
unlabeledCharts?: UnlabeledCharts;
}) {
const [open, setOpen] = useState(false);
const panelId = useId();
Expand Down Expand Up @@ -179,6 +188,8 @@ export function GroupToolbar({
groupSlug={groupSlug}
snapshot={snapshot}
globalSnapshot={globalSnapshot}
seriesLabels={seriesLabels}
unlabeledCharts={unlabeledCharts}
/>
<div className="global-filter-row group-series-row">
<span className="global-filter-label">Series</span>
Expand Down Expand Up @@ -206,8 +217,9 @@ export function GroupToolbar({
data-value={label}
aria-pressed={isActive}
onClick={() => toggleGroupSeries(groupSlug, label)}
title={chipTitle(label, seriesLabels, unlabeledCharts)}
>
{label}
{relabel(label, seriesLabels)}
Comment thread
lwwmanning marked this conversation as resolved.
Resolved
</button>
);
})}
Expand Down Expand Up @@ -262,13 +274,17 @@ function MacroRow({
groupSlug,
snapshot,
globalSnapshot,
seriesLabels,
unlabeledCharts = {},
}: {
label: string;
dim: 'engine' | 'format';
universe: string[];
groupSlug: string;
snapshot: ReturnType<typeof getGroupSnapshot>;
globalSnapshot: ReturnType<typeof getGlobalFilterSnapshot>;
seriesLabels?: SeriesLabels | undefined;
unlabeledCharts?: UnlabeledCharts;
}) {
return (
<div className="global-filter-row group-macro-row">
Expand All @@ -294,11 +310,55 @@ function MacroRow({
data-value={value}
aria-pressed={isActive}
onClick={() => applyGroupMacro(groupSlug, dim, value)}
title={chipTitle(value, seriesLabels, unlabeledCharts)}
>
{value}
{relabel(value, seriesLabels)}
</button>
);
})}
</div>
);
}

/**
* A chip's text: the series label with each segment the group renames (see `SeriesLabels`)
* replaced. The chips otherwise show the real labels, so this is not `displaySeriesLabel`.
*/
function relabel(label: string, labels: SeriesLabels | undefined): string {
if (labels === undefined) {
return label;
}
return label
.split(':')
.map((segment) => labels[segment] ?? segment)
.join(':');
}

/** For each format a group renames, the group's charts that keep the real name. */
type UnlabeledCharts = Readonly<Record<string, readonly string[]>>;

/**
* A renamed chip's title, when some of the group's charts keep the real name: the chip also
* toggles the series there. In the Compression & File Size group, the `parquet (zstd)` chip's
* title is "parquet (zstd), and parquet on clickbench", because ClickBench's Parquet is its
* published Snappy files.
*/
function chipTitle(
label: string,
labels: SeriesLabels | undefined,
unlabeledCharts: UnlabeledCharts,
): string | undefined {
const shown = relabel(label, labels);
const charts = [
...new Set(
label
.split(':')
.filter((segment) => labels?.[segment] !== undefined)
.flatMap((segment) => unlabeledCharts[segment] ?? []),
),
];
if (shown === label || charts.length === 0) {
return undefined;
}
return `${shown}, and ${label} on ${charts.join(', ')}`;
}
Loading
Loading