Repository navigation
FEAT: Scanner dataset selection: required, linked and first datasets (-5.5) - #4476
Conversation
…(-5.5) Carves the generation-independent scanner and dataset changes out of the GUI generation layer (microsoft#4135), so they can be reviewed on their own: - ScenarioDatasetOptions.min_selected, validated against max_selected and enforced when a request selects datasets ("Select a dataset." / "Select at least N datasets."). foundry.assert_policy declares 1. - Scenario.get_dataset_options() reports inline seeds/seed_groups as fixed sources with no required selection. - Scanner launch page: Browse datasets moves into the Datasets header, the Datasets heading and picker are marked required, a launch without a required dataset reports it on the focused picker, a scenario with no default datasets can choose its first one, and Reset datasets shows only after source or per-dataset limit edits. - Repeatable ?dataset= links preselect datasets exactly as named; a scanner that takes no datasets blocks the launch with a link that drops them. - Estimate and launch errors append the server's 422 field messages. - Dataset catalog gains an Origins facet persisted as origin URL parameters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
An incompatible dataset link can leave a fixed-source scenario with a backend-rejected selection and no available removal, reset, or recovery action.
1 open issue
What changed in this PR
This PR improves dataset selection in PyRIT’s Scanner, allowing scenarios without defaults to select their first dataset and supporting dataset preselection through launch links.
Changes:
- Adds minimum dataset requirements and preserves fixed sources for inline-seed scenarios.
- Adds linked dataset selection, picker validation, and clearer API error messages.
- Adds dataset-origin filtering and a shared Browse datasets action.
| File | Description |
|---|---|
| tests/unit/scenario/foundry/test_assert_policy.py | Tests required dataset selection. |
| tests/unit/scenario/core/test_scenario_dataset_overrides.py | Tests selection bounds and inline-source restrictions. |
| pyrit/scenario/scenarios/foundry/assert_policy.py | Declares the minimum dataset requirement. |
| pyrit/scenario/core/scenario.py | Reports inline sources as fixed. |
| pyrit/scenario/core/dataset_overrides.py | Enforces minimum dataset counts. |
| pyrit/models/scenario_dataset.py | Adds validated minimum-selection metadata. |
| frontend/src/utils/routeParams.ts | Parses repeatable dataset link parameters. |
| frontend/src/utils/routeParams.test.ts | Tests exact dataset-name parsing. |
| frontend/src/types/index.ts | Exposes minimum-selection metadata. |
| frontend/src/services/errors.ts | Formats API field-validation messages. |
| frontend/src/services/errors.test.ts | Tests validation-message formatting. |
| frontend/src/hooks/useScenarioRunEstimate.ts | Displays detailed estimate errors. |
| frontend/src/hooks/useScenarioLaunch.ts | Displays detailed launch errors. |
| frontend/src/hooks/useScenarioLaunch.test.tsx | Tests launch validation errors. |
| frontend/src/components/Scenarios/scenarioLaunchRequest.ts | Validates required and unsupported linked datasets. |
| frontend/src/components/Scenarios/scenarioLaunchRequest.test.ts | Tests dataset request validation. |
| frontend/src/components/Scenarios/scenarioLaunchForm.ts | Seeds dataset sources from links. |
| frontend/src/components/Scenarios/ScenarioDetail.tsx | Integrates linked datasets, required selection, and recovery. |
| frontend/src/components/Scenarios/ScenarioDetail.test.tsx | Tests dataset launch workflows. |
| frontend/src/components/Scenarios/ScenarioDetail.styles.ts | Styles section-header actions. |
| frontend/src/components/Scenarios/ScenarioDatasetTable.tsx | Enables first selection and picker-level errors. |
| frontend/src/components/Scenarios/ScenarioDatasetTable.test.tsx | Tests empty, fixed, and subset selections. |
| frontend/src/components/Scenarios/DatasetActions.tsx | Adds the shared catalog navigation action. |
| frontend/src/components/Scenarios/DatasetActions.test.tsx | Tests catalog navigation. |
| frontend/src/components/Scenarios/DatasetActions.styles.ts | Styles dataset actions and touch targets. |
| frontend/src/components/Datasets/datasetQuery.ts | Adds URL-backed origin filtering. |
| frontend/src/components/Datasets/datasetQuery.test.ts | Tests origin filtering and serialization. |
| frontend/src/components/Datasets/DatasetCatalog.tsx | Adds the Origins filter control. |
| frontend/src/components/Datasets/DatasetCatalog.test.tsx | Tests origin-filter interaction. |
| frontend/e2e/numeric-controls.spec.ts | Updates reset-button expectations. |
| doc/gui/0_gui.md | Documents dataset selection, links, and origins. |
| .github/instructions/scenarios.instructions.md | Documents dataset capability declarations. |
Review effort: Balanced
Enforce a scenario's declared min_selected in the library for every named selection, with or without dataset overrides, and remove the frontend and AssertPolicy copies of the rule so the GUI shows the server's message. Block linked datasets on scanners with fixed sources, open Browse datasets in a new tab from one shared action, show Reset datasets only for real edits, name the field in validation messages, mark a required dataset once, share origin labels, show the dataset minimum in the CLI listing, and keep the docs and instructions accurate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scenario that declares min_selected and has no default datasets has no default run to size. Return an unavailable default estimate with the selection message instead of raising, so the catalog shows the reason without logging a failure on every load. Configured estimates and runs still reject the selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aunch - Size the default run against the default configuration's dataset options, so a reused scenario instance returns the unavailable estimate instead of raising. - Label a 400/422 estimate as a configuration to complete, not a backend failure. - Focus the launch error banner after a rejected launch so its reason scrolls into view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Launch scan with fewer datasets than the scenario declares in min_selected now keeps the form open and shows the server's wording on the focused, required Add dataset picker instead of opening the Run preview. The text is the server's estimate rejection for this request when there is one, else the library's validate_minimum_selection wording; the library stays the only enforcer. The message clears once enough datasets are selected. The Run preview disables Launch scan while the server rejected the current request's estimate (HTTP 400 or 422), and those rejections are labelled "Fix the configuration to request an estimate." The missing-input label stays for configurations the form can't build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The server checks parameters and techniques before the dataset count, so its estimate rejection can be about another field. The picker now always shows the library's wording for the declared minimum; other rejections stay in the run estimate and disable the preview's Launch scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
Review of head caa9d1a. The code itself looks right to me — one owner for the min-selected rule, server-authoritative validation, linked names passed through byte-exact — and the touched Python suites pass locally (256/256). My one real pre-merge ask is non-code: the PR description and screenshots still show the UI from before 77b7a33. On the current head there is no "No dataset selected." empty state, no "Select a dataset." message on the picker, no focus move to it, and no required marker on the section header — the server's message shows in the run estimate and, after a confirmed launch, in the top-of-form error bar with focus on the Launch button. Since this is the ready-for-review insert of the stack, please refresh the description/screenshots to match what merges (and in doing so confirm the "no client-side block" UX is the intended endpoint, since the description currently promises the opposite). The failing Build latest check is the docs link checker hitting 503s on pre-existing external links — a re-run should clear it.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The description and screenshots were refreshed for db15b2f in the earlier repair pass. They now show the required picker, its focused inline error without a Run preview, and the updated error/recovery states. The intended endpoint is the picker check before preview, not the earlier no-client-block behavior described at caa9d1a. This follow-up adds the requested CLI dataset guidance and integrates main; it makes no further UI changes. Verified on published commit |
Merge main 830f745, which includes microsoft#4476 (-5.5), against a virtual base of 3e0820b plus microsoft#4476's initial carve (60be4e9), so main's reviewed scanner and dataset behavior wins and this branch keeps only its generation additions. - DatasetActions: main's new-tab Browse datasets plus this branch's Generate datasets, now in its own component so callers without a scenario (the preset editor main added) show Browse only and don't load generators. - DatasetCatalog: keep main's shared origin labels and this branch's Generate datasets header button. - ScenarioDetail tests: keep main's new tests; the Datasets header test checks both the new-tab Browse link and the Generate link. - Docs: keep main's Scanner dataset paragraph and Origins sentence; add the Generate datasets section. - prompt-generation e2e: follow main's origin labels ("Generated") and its empty selection, which shows only the picker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Part -5.5 of the ASSERT stack. Design and how to use it: ASSERT in PyRIT.
This layer holds the scanner and dataset changes from #4135 that don't need dataset generation, so they can be reviewed on their own. It targets
maindirectly; #4135 will be rebuilt on top of it.Description
A scenario can now say it needs at least one dataset, and the scanner launch page lets you choose the first dataset, link datasets in by URL, and see why a launch was rejected. On
main,foundry.assert_policyhas no default datasets and the launch page offers no way to pick one, so it can't be launched from the GUI.ScenarioDatasetOptionsgainsmin_selected, the counterpart ofmax_selected. It must be positive and no larger thanmax_selected.Scenarioenforces it for every named selection, with or without dataset overrides, with "Select a dataset." / "Select at least N datasets.", so estimates and runs report that message early.foundry.assert_policydeclaresmin_selected=1instead of checking for datasets itself. The value reaches the GUI through the existing catalog and estimate models, andpyrit_scan list-scenariosshows it.Scenario.get_dataset_options()reports inlineseeds/seed_groupsasfixedsources, because named sources can't replace them, and drops any declared minimum. Among registered scenarios this affects onlytechnique.execute, which takes no dataset input.min_selectedmarks its dataset picker required; until a dataset is chosen, the run estimate shows the server's "Select a dataset." after "Fix the configuration to request an estimate.", and Launch scan shows it on the focused picker instead of opening the Run preview.?dataset=parameters preselect datasets exactly as named, replacing the default or preset datasets; other dataset settings, such as limits and filters, stay. The server validates the names. On a scanner that takes no datasets or has fixed sources, linked names block the launch with a link to open the page without them.originparameters. The scanner's dataset picker uses the same origin names.foundry.assert_policybefore, onmain: no dataset picker, so it can't be launched.After: the dataset picker is required, with Browse datasets in the header.
After, Launch scan without a dataset: the server's "Select a dataset." on the focused, required picker, with no Run preview.
After, opened with
?dataset=jailbreak_templates: the dataset is preselected.More screenshots
Ordinary scanner (
foundry.red_team_agent), before onmain: Browse datasets is a link under the table, and Reset datasets shows on untouched defaults.After, untouched: Browse datasets in the header, no Reset datasets.
After adding a dataset: Reset datasets appears.
Blank linked name (
foundry.red_team_agent?dataset=%20): the run estimate shows the server's message.Scanner that takes no datasets (
technique.execute?dataset=jailbreak_templates): the launch is blocked with a link that drops the linked datasets.Dataset catalog: the Origins filter, then filtered to Local files.
foundry.assert_policyDatasets section at 390 px:Screenshots use the repo's offline end-to-end backend with no provider or model calls. Launch-page controls outside the Datasets section aren't from this PR.