Repository navigation
FEAT Exact dataset example selection contract (backend, #2757 part 1) - #4099
Julio César Suástegui (juliosuas) wants to merge 21 commits into
Conversation
Add a bounded example_selection contract whose fingerprint covers dataset membership and stored SeedRecord content. Estimate, launch, and resume share one memory-backed resolver, and named-dataset requests stay unchanged when the field is omitted. Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
Resolution and estimates keep every stored member. Requiring exactly one objective is execution policy for microsoft#2760, so this selection contract no longer rejects or relabels that shape. Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
Resolution and estimates still accept any stored objective count. When launch constructs groups and that construction fails, return unsupported with the example id and the reason instead of an untyped constructor error. Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical findings allow incompatible scenarios to fail without typed errors, changed selections to bypass resume checks, and persisted conditions to be silently dropped.
4 open issues
What changed in this PR
Adds the backend exact dataset-selection contract for #2757, connecting dataset browsing to scenario estimates, launch, and resume. Frontend selection and execution-policy work remain separate.
Changes:
- Adds bounded, dataset-scoped references with membership/content fingerprints and typed errors.
- Resolves selected examples through canonical memory reads without default caps or resampling.
- Persists selections for resume and adds resolver, HTTP, and SQL compatibility coverage.
| File | Description |
|---|---|
tests/unit/backend/test_seed_example_routes.py |
Checks list/detail fingerprint consistency. |
tests/unit/backend/test_dataset_example_selection.py |
Tests resolution, freshness, limits, and persistence. |
tests/unit/backend/test_dataset_example_selection_routes.py |
Tests HTTP estimates, launch preparation, and errors. |
tests/integration/memory/test_seed_examples_azure_sql_integration.py |
Adds Azure SQL batch-read assertions. |
pyrit/scenario/core/scenario.py |
Uses resolved selection counts in estimates. |
pyrit/scenario/core/dataset_configuration.py |
Adds unsampled exact-example configuration. |
pyrit/scenario/core/__init__.py |
Exports the exact-example configuration. |
pyrit/scenario/__init__.py |
Exposes the configuration publicly. |
pyrit/models/scenario_dataset.py |
Defines selection references, bounds, and errors. |
pyrit/models/catalog/scenario.py |
Adds selection to shared scenario requests. |
pyrit/models/__init__.py |
Exports selection models and errors. |
pyrit/memory/memory_interface.py |
Adds dataset-scoped batch example reads. |
pyrit/backend/services/scenario_service.py |
Integrates selection into estimate/load handling. |
pyrit/backend/services/scenario_run_service.py |
Resolves and persists launch selections. |
pyrit/backend/services/scenario_configuration_resolver.py |
Integrates asynchronous exact-selection resolution. |
pyrit/backend/services/dataset_service.py |
Includes fingerprints in browse responses. |
pyrit/backend/services/dataset_example_selection.py |
Resolves references and checks freshness/conflicts. |
pyrit/backend/services/dataset_example_fingerprint.py |
Implements selection-key parsing and fingerprints. |
pyrit/backend/routes/scenarios.py |
Maps selection failures to HTTP responses. |
pyrit/backend/models/datasets.py |
Adds fingerprints to example response models. |
Review effort: Balanced
Opt out scenarios that read named datasets or a private configuration, and reject them before memory reads or initialization. Resume re-resolves a saved selection and rejects a different one. Non-objective conditions and group construction stay typed and off the event loop. Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
…crosoft#2757) - Treat a stored empty condition list as no conditions. Only nonempty conditions on a non-objective seed are rejected as unsupported. - Reject an exact launch or estimate when a selected technique would skip a selected example or run it only after objective-only adaptation. The check reuses prepare_seed_group and the factory's group validation, so it cannot drift from the scenario's skip/adaptation handling. - Regression tests for None, empty, and nonempty conditions, and for role_play_movie_script and many_shot under RapidResponse.
…lection-backend-v2 Resolve import conflicts in dataset_configuration.py and scenario.py: keep ExactExampleDatasetConfiguration and Mapping alongside main's DatasetSource, DatasetSourceKind, and Awaitable (required data, microsoft#3464).
Resolve conflicts with microsoft#3794 (configure scan targets before launch): - scenario_configuration_resolver: keep the exact-selection imports and the readiness/adversarial-override imports; resolve_configuration_async now accepts and forwards scoring_target_name so launch, estimate and dataset load keep the explicit scoring target with an exact selection. - scenario_run_service: optional launch fields keep both scoring_target_name and example_selection. - tests: patch resolve_target on the resolver class, as main's tests do, and cover scoring_target_name together with an exact selection.
LeakReplay (microsoft#3689, merged to main) only accepts LeakReplayDatasetConfiguration, so an exact selection would be advertised and then rejected at launch.
…crosoft#2757) The exact-selection compatibility check built its own factory map in the resolver, letting a technique enum's catalog defaults replace registered factories of the same name. TextAdaptive (and the base readiness view) give registered factories precedence, so the check could validate a different factory from the one that executes: a registered objective-only crescendo accepted a conversation that initialization then skipped. - Add Scenario._get_selected_technique_factories, which resolves the selection with _resolve_scenario_techniques and looks factories up in _get_readiness_factories, the view readiness and execution share. - The resolver configures its introspection scenario with the run's scenario_params and resolved values, then checks those factories. Estimate and launch pass scenario_params through. - Regression: real TextAdaptive with a registered objective-only crescendo override rejects a conversation plus objective as unsupported.
…crosoft#2757) The technique compatibility check also ran during configured estimates and rebuilt every selected group, so a stored simulated conversation was reconstructed as SeedSimulatedConversation and read its legacy YAML prompt paths. microsoft#2757 requires estimates not to load legacy templates. - validate_technique_factories(inspect_only=True) builds the groups from stored inspection data. A simulated conversation becomes an unvalidated placeholder carrying its stored type, group, num_turns and sequence (model_construct), so no legacy path or default template is opened. The shared prepare/require_unchanged checks still run on that group. - ScenarioService estimates pass inspect_exact_selection=True. Launch keeps full reconstruction and validation of the executable seeds. - Regressions through ScenarioService.estimate_scenario_run_size_async with real RapidResponse, crescendo, an explicit scorer and a stored legacy simulated-conversation record: estimate count 1 with zero legacy-loader and template calls, and many_shot is still rejected as unsupported.
Preserve objective decomposition and technique flags when rebuilding stored seeds. Resolve exact groups through the shared dataset seam and reject population-changing overrides. Opt ASSERT out of REST exact selection until its selective tester and baseline policies can preserve the selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve conflicts with microsoft#4130 (trace-grounded ASSERT judging): keep main's v3 judge and VERSION 3, keep AssertPolicy.SUPPORTS_EXACT_EXAMPLE_SELECTION = False, and keep the exact-selection test on top of main's rewritten ASSERT tests.
# Conflicts: # pyrit/backend/README.md
Resolve tests/unit/scenario/foundry/test_assert_policy.py by keeping main's ASSERT metrics intent from microsoft#4131 plus the PR's exact selection imports and tests. microsoft#4371 landed cleanly in tests/unit/backend/test_chat_scoring.py.
|
Heads-up on an overlap with #4350, which changes how resume works. #4350 stores each run's planned seed groups when the run is created ( This PR goes the other way for exact selections. On resume, Suggestion: keep the fingerprint check for preview, estimate, and launch, where it guards against running on content the user didn't see, and drop it from the resume path. Both PRs also touch |
|
Thanks Richard Lundeen (@richlundeen), agreed. Once #4350 is in, resume should rebuild from the stored Plan:
I'll take care of reconciling |
Resume now rebuilds from the seed groups microsoft#4350 stores with the run, so it no longer re-resolves an exact selection against live datasets. Preview, estimate and launch keep the fingerprint check.
|
Richard Lundeen (@richlundeen) I merged main (with #4350) into this branch. Resume now relies on #4350's stored
The only textual conflict was the |


Part 1 of #2757: backend exact-selection contract. Frontend follows in a separate PR.
What
selection_key+example_id, with a fingerprint covering membership and content.get_seed_examples_by_ids_async.stale.Objective-count policy is left to #2760: the resolver and estimate accept any stored objective count. If launch can't build an attack group from the selected members, it returns a typed
unsupportederror (400) naming the example, not a 500.Out of scope: #2760 (execution policy), #2759 (frontend multi-select).
Testing
tests/unit/backend: 3600 passed, 4 skipped (external credentials).tests/unit/memory: 1986 passed, 1 skipped (Azure SQL).tests/unit/scenario: 2210 passed. Dataset suites: 5425 passed.AttackSeedGroup(0/2 objectives, multiple simulated conversations, HTTP 400 path).example_selectionunchanged.AI-assisted; reviewed and tested by me.