Skip to content

FEAT: Scanner dataset selection: required, linked and first datasets (-5.5) - #4476

Merged
Richard Lundeen (richlundeen) merged 9 commits into
microsoft:mainfrom
richlundeen:richlundeen-scanner-dataset-selection
Nov 21, 2026
Merged

Richard Lundeen (richlundeen) merged 9 commits into
microsoft:mainfrom
richlundeen:richlundeen-scanner-dataset-selection

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Part -5.5 of the ASSERT stack. Design and how to use it: ASSERT in PyRIT.

PR
-1 #4132 Judge dimensions, presets and shared techniques merged
-2 #4130 Trace evidence (OpenTelemetry) merged
-3 #4131 Headline metrics merged
-4 #4133 Staged generation (SDK) merged
-5 #4134 Generation jobs (backend) merged
-5.5 #4476 Scanner dataset selection this PR
-6 #4135 Generate from the GUI
-7 #4144 In-process agents (docs)
-8 #4180 Scenario-aware dataset compatibility
-9 #4226 ASSERT results grouped by generated category
-10 #4228 Background dataset loading (all scanners)

This layer holds the scanner and dataset changes from #4135 that don't need dataset generation, so they can be reviewed on their own. It targets main directly; #4135 will be rebuilt on top of it.

Description

A scenario can now say it needs at least one dataset, and the scanner launch page lets you choose the first dataset, link datasets in by URL, and see why a launch was rejected. On main, foundry.assert_policy has no default datasets and the launch page offers no way to pick one, so it can't be launched from the GUI.

  • Required datasets (library): ScenarioDatasetOptions gains min_selected, the counterpart of max_selected. It must be positive and no larger than max_selected. Scenario enforces it for every named selection, with or without dataset overrides, with "Select a dataset." / "Select at least N datasets.", so estimates and runs report that message early. foundry.assert_policy declares min_selected=1 instead of checking for datasets itself. The value reaches the GUI through the existing catalog and estimate models, and pyrit_scan list-scenarios shows it.
  • Inline-only scenarios (library): Scenario.get_dataset_options() reports inline seeds / seed_groups as fixed sources, because named sources can't replace them, and drops any declared minimum. Among registered scenarios this affects only technique.execute, which takes no dataset input.
  • Launch page, Datasets section:
    • Browse datasets moves from a link under the table into the section header and opens the catalog in a new tab, so the form keeps its edits. The preset editor uses the same action.
    • A scenario with no default datasets can choose its first one. A scenario that declares min_selected marks its dataset picker required; until a dataset is chosen, the run estimate shows the server's "Select a dataset." after "Fix the configuration to request an estimate.", and Launch scan shows it on the focused picker instead of opening the Run preview.
    • Reset datasets appears only once the datasets or per-dataset limit are changed, since that's all it resets.
  • Linked datasets: repeatable ?dataset= parameters preselect datasets exactly as named, replacing the default or preset datasets; other dataset settings, such as limits and filters, stay. The server validates the names. On a scanner that takes no datasets or has fixed sources, linked names block the launch with a link to open the page without them.
  • Error messages: scanner estimate and launch errors include the server's validation messages, with the field they are about, for example "Request validation failed: datasets.sources.0.name: Dataset names must be non-empty." instead of only "Request validation failed". An estimate the server rejects reads "Fix the configuration to request an estimate." with that message, and the Run preview doesn't offer Launch scan for it.
  • Dataset catalog: an Origins filter (local files, remote provider, generated, …), kept in the URL as repeated origin parameters. The scanner's dataset picker uses the same origin names.
  • The GUI doc describes the new behavior.

foundry.assert_policy before, on main: no dataset picker, so it can't be launched.

Before: foundry.assert_policy Datasets section on main says the scenario has no named datasets and offers no picker

After: the dataset picker is required, with Browse datasets in the header.

After: foundry.assert_policy with Browse datasets in the header and the required Add dataset picker

After, Launch scan without a dataset: the server's "Select a dataset." on the focused, required picker, with no Run preview.

After: Launch scan without a dataset keeps the form open and shows Select a dataset inline on the focused, required Add dataset picker, with no Run preview

After, opened with ?dataset=jailbreak_templates: the dataset is preselected.

After: foundry.assert_policy opened with a linked dataset, which is preselected

More screenshots

Ordinary scanner (foundry.red_team_agent), before on main: Browse datasets is a link under the table, and Reset datasets shows on untouched defaults.

Before: foundry.red_team_agent Datasets section with the Browse datasets link under the table and Reset datasets on untouched defaults

After, untouched: Browse datasets in the header, no Reset datasets.

After: foundry.red_team_agent Datasets section with Browse datasets in the header and no Reset datasets

After adding a dataset: Reset datasets appears.

After: foundry.red_team_agent Datasets section after adding a dataset, with Reset datasets

Blank linked name (foundry.red_team_agent?dataset=%20): the run estimate shows the server's message.

Run estimate showing Request validation failed: datasets.sources.0.name: Dataset names must be non-empty

Scanner that takes no datasets (technique.execute?dataset=jailbreak_templates): the launch is blocked with a link that drops the linked datasets.

technique.execute with a linked dataset: blocking error with Open without the linked datasets and Launch scan disabled

Dataset catalog: the Origins filter, then filtered to Local files.

Dataset catalog with the Origins filter open

Dataset catalog filtered to the Local files origin

foundry.assert_policy Datasets section at 390 px:

foundry.assert_policy Datasets section at 390 px

Screenshots use the repo's offline end-to-end backend with no provider or model calls. Launch-page controls outside the Datasets section aren't from this PR.

…(-5.5)

Carves the generation-independent scanner and dataset changes out of the GUI
generation layer (microsoft#4135), so they can be reviewed on their own:

- ScenarioDatasetOptions.min_selected, validated against max_selected and
  enforced when a request selects datasets ("Select a dataset." /
  "Select at least N datasets."). foundry.assert_policy declares 1.
- Scenario.get_dataset_options() reports inline seeds/seed_groups as fixed
  sources with no required selection.
- Scanner launch page: Browse datasets moves into the Datasets header, the
  Datasets heading and picker are marked required, a launch without a
  required dataset reports it on the focused picker, a scenario with no
  default datasets can choose its first one, and Reset datasets shows only
  after source or per-dataset limit edits.
- Repeatable ?dataset= links preselect datasets exactly as named; a scanner
  that takes no datasets blocks the launch with a link that drops them.
- Estimate and launch errors append the server's 422 field messages.
- Dataset catalog gains an Origins facet persisted as origin URL parameters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 17:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An incompatible dataset link can leave a fixed-source scenario with a backend-rejected selection and no available removal, reset, or recovery action.

1 open issue
What changed in this PR

This PR improves dataset selection in PyRIT’s Scanner, allowing scenarios without defaults to select their first dataset and supporting dataset preselection through launch links.

Changes:

  • Adds minimum dataset requirements and preserves fixed sources for inline-seed scenarios.
  • Adds linked dataset selection, picker validation, and clearer API error messages.
  • Adds dataset-origin filtering and a shared Browse datasets action.
File Description
tests/​unit/​scenario/​foundry/​test_assert_policy.py Tests required dataset selection.
tests/​unit/​scenario/​core/​test_scenario_dataset_overrides.py Tests selection bounds and inline-source restrictions.
pyrit/​scenario/​scenarios/​foundry/​assert_policy.py Declares the minimum dataset requirement.
pyrit/​scenario/​core/​scenario.py Reports inline sources as fixed.
pyrit/​scenario/​core/​dataset_overrides.py Enforces minimum dataset counts.
pyrit/​models/​scenario_dataset.py Adds validated minimum-selection metadata.
frontend/​src/​utils/​routeParams.ts Parses repeatable dataset link parameters.
frontend/​src/​utils/​routeParams.test.ts Tests exact dataset-name parsing.
frontend/​src/​types/​index.ts Exposes minimum-selection metadata.
frontend/​src/​services/​errors.ts Formats API field-validation messages.
frontend/​src/​services/​errors.test.ts Tests validation-message formatting.
frontend/​src/​hooks/​useScenarioRunEstimate.ts Displays detailed estimate errors.
frontend/​src/​hooks/​useScenarioLaunch.ts Displays detailed launch errors.
frontend/​src/​hooks/​useScenarioLaunch.test.tsx Tests launch validation errors.
frontend/​src/​components/​Scenarios/​scenarioLaunchRequest.ts Validates required and unsupported linked datasets.
frontend/​src/​components/​Scenarios/​scenarioLaunchRequest.test.ts Tests dataset request validation.
frontend/​src/​components/​Scenarios/​scenarioLaunchForm.ts Seeds dataset sources from links.
frontend/​src/​components/​Scenarios/​ScenarioDetail.tsx Integrates linked datasets, required selection, and recovery.
frontend/​src/​components/​Scenarios/​ScenarioDetail.test.tsx Tests dataset launch workflows.
frontend/​src/​components/​Scenarios/​ScenarioDetail.styles.ts Styles section-header actions.
frontend/​src/​components/​Scenarios/​ScenarioDatasetTable.tsx Enables first selection and picker-level errors.
frontend/​src/​components/​Scenarios/​ScenarioDatasetTable.test.tsx Tests empty, fixed, and subset selections.
frontend/​src/​components/​Scenarios/​DatasetActions.tsx Adds the shared catalog navigation action.
frontend/​src/​components/​Scenarios/​DatasetActions.test.tsx Tests catalog navigation.
frontend/​src/​components/​Scenarios/​DatasetActions.styles.ts Styles dataset actions and touch targets.
frontend/​src/​components/​Datasets/​datasetQuery.ts Adds URL-backed origin filtering.
frontend/​src/​components/​Datasets/​datasetQuery.test.ts Tests origin filtering and serialization.
frontend/​src/​components/​Datasets/​DatasetCatalog.tsx Adds the Origins filter control.
frontend/​src/​components/​Datasets/​DatasetCatalog.test.tsx Tests origin-filter interaction.
frontend/​e2e/​numeric-controls.spec.ts Updates reset-button expectations.
doc/​gui/​0_gui.md Documents dataset selection, links, and origins.
.github/​instructions/​scenarios.instructions.md Documents dataset capability declarations.

Review effort: Balanced

Enforce a scenario's declared min_selected in the library for every named
selection, with or without dataset overrides, and remove the frontend and
AssertPolicy copies of the rule so the GUI shows the server's message.
Block linked datasets on scanners with fixed sources, open Browse datasets
in a new tab from one shared action, show Reset datasets only for real
edits, name the field in validation messages, mark a required dataset once,
share origin labels, show the dataset minimum in the CLI listing, and keep
the docs and instructions accurate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scenario that declares min_selected and has no default datasets has no
default run to size. Return an unavailable default estimate with the
selection message instead of raising, so the catalog shows the reason
without logging a failure on every load. Configured estimates and runs
still reject the selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aunch

- Size the default run against the default configuration's dataset options, so a reused scenario instance returns the unavailable estimate instead of raising.
- Label a 400/422 estimate as a configuration to complete, not a backend failure.
- Focus the launch error banner after a rejected launch so its reason scrolls into view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Launch scan with fewer datasets than the scenario declares in min_selected
now keeps the form open and shows the server's wording on the focused,
required Add dataset picker instead of opening the Run preview. The text is
the server's estimate rejection for this request when there is one, else
the library's validate_minimum_selection wording; the library stays the
only enforcer. The message clears once enough datasets are selected.

The Run preview disables Launch scan while the server rejected the current
request's estimate (HTTP 400 or 422), and those rejections are labelled
"Fix the configuration to request an estimate." The missing-input label
stays for configurations the form can't build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The server checks parameters and techniques before the dataset count, so its
estimate rejection can be about another field. The picker now always shows
the library's wording for the declared minimum; other rejections stay in the
run estimate and disable the preview's Launch scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head caa9d1a. The code itself looks right to me — one owner for the min-selected rule, server-authoritative validation, linked names passed through byte-exact — and the touched Python suites pass locally (256/256). My one real pre-merge ask is non-code: the PR description and screenshots still show the UI from before 77b7a33. On the current head there is no "No dataset selected." empty state, no "Select a dataset." message on the picker, no focus move to it, and no required marker on the section header — the server's message shows in the run estimate and, after a confirmed launch, in the top-of-form error bar with focus on the Launch button. Since this is the ready-for-review insert of the stack, please refresh the description/screenshots to match what merges (and in doing so confirm the "no client-side block" UX is the intended endpoint, since the description currently promises the opposite). The failing Build latest check is the docs link checker hitting 503s on pre-existing external links — a re-run should clear it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

review:5998815736

The description and screenshots were refreshed for db15b2f in the earlier repair pass. They now show the required picker, its focused inline error without a Run preview, and the updated error/recovery states. The intended endpoint is the picker check before preview, not the earlier no-client-block behavior described at caa9d1a. This follow-up adds the requested CLI dataset guidance and integrates main; it makes no further UI changes.

Verified on published commit a77d58b50daf3708d553d6187b2315f44273ecca.

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 21, 2026
Merged via the queue into microsoft:main with commit 7bb890f Nov 21, 2026
65 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-scanner-dataset-selection branch November 21, 2026 00:20
Richard Lundeen (richlundeen) added a commit to richlundeen/PyRIT that referenced this pull request Nov 21, 2026
Merge main 830f745, which includes microsoft#4476 (-5.5), against a virtual base of
3e0820b plus microsoft#4476's initial carve (60be4e9), so main's reviewed scanner and
dataset behavior wins and this branch keeps only its generation additions.

- DatasetActions: main's new-tab Browse datasets plus this branch's Generate
  datasets, now in its own component so callers without a scenario (the preset
  editor main added) show Browse only and don't load generators.
- DatasetCatalog: keep main's shared origin labels and this branch's Generate
  datasets header button.
- ScenarioDetail tests: keep main's new tests; the Datasets header test checks
  both the new-tab Browse link and the Generate link.
- Docs: keep main's Scanner dataset paragraph and Origins sentence; add the
  Generate datasets section.
- prompt-generation e2e: follow main's origin labels ("Generated") and its empty
  selection, which shows only the picker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants