Skip to content

FEAT Exact dataset example selection contract (backend, #2757 part 1) - #4099

Open
Julio César Suástegui (juliosuas) wants to merge 21 commits into
microsoft:mainfrom
juliosuas:feat/2757-exact-selection-backend-v2
Open

Julio César Suástegui (juliosuas) wants to merge 21 commits into
microsoft:mainfrom
juliosuas:feat/2757-exact-selection-backend-v2

Conversation

@juliosuas

Copy link
Copy Markdown
Contributor

Part 1 of #2757: backend exact-selection contract. Frontend follows in a separate PR.

What

  • Dataset-scoped selection_key + example_id, with a fingerprint covering membership and content.
  • One read-only resolver shared by preview, estimate and launch, so all three see the same rows.
  • Canonical memory read get_seed_examples_by_ids_async.
  • Explicit typed errors: stale (409), duplicate, missing, conflicting, unsupported (400). Selection over the shared 100-item list limit returns 422 instead of truncating.
  • An exact selection is used as-is: no default cap of 5 and no resampling.
  • Resume re-resolves the stored references and fingerprints; a changed row returns stale.
  • Named datasets and scorers behave as before when no selection is given.

Objective-count policy is left to #2760: the resolver and estimate accept any stored objective count. If launch can't build an attack group from the selected members, it returns a typed unsupported error (400) naming the example, not a 500.

Out of scope: #2760 (execution policy), #2759 (frontend multi-select).

Testing

  • tests/unit/backend: 3600 passed, 4 skipped (external credentials).
  • tests/unit/memory: 1986 passed, 1 skipped (Azure SQL).
  • tests/unit/scenario: 2210 passed. Dataset suites: 5425 passed.
  • New launch tests use the real AttackSeedGroup (0/2 objectives, multiple simulated conversations, HTTP 400 path).
  • Regression without example_selection unchanged.
  • ruff check/format and ty on changed files pass.
  • Not run: Azure SQL integration test, e2e.

AI-assisted; reviewed and tested by me.

Add a bounded example_selection contract whose fingerprint covers dataset
membership and stored SeedRecord content. Estimate, launch, and resume share
one memory-backed resolver, and named-dataset requests stay unchanged when
the field is omitted.

Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
Resolution and estimates keep every stored member. Requiring exactly one
objective is execution policy for microsoft#2760, so this selection contract no longer
rejects or relabels that shape.

Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
Resolution and estimates still accept any stored objective count. When launch
constructs groups and that construction fails, return unsupported with the
example id and the reason instead of an untyped constructor error.

Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 9, 2026 07:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical findings allow incompatible scenarios to fail without typed errors, changed selections to bypass resume checks, and persisted conditions to be silently dropped.

4 open issues
What changed in this PR

Adds the backend exact dataset-selection contract for #2757, connecting dataset browsing to scenario estimates, launch, and resume. Frontend selection and execution-policy work remain separate.

Changes:

  • Adds bounded, dataset-scoped references with membership/content fingerprints and typed errors.
  • Resolves selected examples through canonical memory reads without default caps or resampling.
  • Persists selections for resume and adds resolver, HTTP, and SQL compatibility coverage.
File Description
tests/​unit/​backend/​test_seed_example_routes.py Checks list/detail fingerprint consistency.
tests/​unit/​backend/​test_dataset_example_selection.py Tests resolution, freshness, limits, and persistence.
tests/​unit/​backend/​test_dataset_example_selection_routes.py Tests HTTP estimates, launch preparation, and errors.
tests/​integration/​memory/​test_seed_examples_azure_sql_integration.py Adds Azure SQL batch-read assertions.
pyrit/​scenario/​core/​scenario.py Uses resolved selection counts in estimates.
pyrit/​scenario/​core/​dataset_configuration.py Adds unsampled exact-example configuration.
pyrit/​scenario/​core/​__init__.py Exports the exact-example configuration.
pyrit/​scenario/​__init__.py Exposes the configuration publicly.
pyrit/​models/​scenario_dataset.py Defines selection references, bounds, and errors.
pyrit/​models/​catalog/​scenario.py Adds selection to shared scenario requests.
pyrit/​models/​__init__.py Exports selection models and errors.
pyrit/​memory/​memory_interface.py Adds dataset-scoped batch example reads.
pyrit/​backend/​services/​scenario_service.py Integrates selection into estimate/load handling.
pyrit/​backend/​services/​scenario_run_service.py Resolves and persists launch selections.
pyrit/​backend/​services/​scenario_configuration_resolver.py Integrates asynchronous exact-selection resolution.
pyrit/​backend/​services/​dataset_service.py Includes fingerprints in browse responses.
pyrit/​backend/​services/​dataset_example_selection.py Resolves references and checks freshness/conflicts.
pyrit/​backend/​services/​dataset_example_fingerprint.py Implements selection-key parsing and fingerprints.
pyrit/​backend/​routes/​scenarios.py Maps selection failures to HTTP responses.
pyrit/​backend/​models/​datasets.py Adds fingerprints to example response models.

Review effort: Balanced

Opt out scenarios that read named datasets or a private configuration, and
reject them before memory reads or initialization. Resume re-resolves a saved
selection and rejects a different one. Non-objective conditions and group
construction stay typed and off the event loop.

Co-authored-by: Julio César Suástegui <juliosuas@users.noreply.github.com>
…crosoft#2757)

- Treat a stored empty condition list as no conditions. Only nonempty
  conditions on a non-objective seed are rejected as unsupported.
- Reject an exact launch or estimate when a selected technique would skip
  a selected example or run it only after objective-only adaptation. The
  check reuses prepare_seed_group and the factory's group validation, so it
  cannot drift from the scenario's skip/adaptation handling.
- Regression tests for None, empty, and nonempty conditions, and for
  role_play_movie_script and many_shot under RapidResponse.
…lection-backend-v2

Resolve import conflicts in dataset_configuration.py and scenario.py: keep ExactExampleDatasetConfiguration and Mapping alongside main's DatasetSource, DatasetSourceKind, and Awaitable (required data, microsoft#3464).
Resolve conflicts with microsoft#3794 (configure scan targets before launch):
- scenario_configuration_resolver: keep the exact-selection imports and
  the readiness/adversarial-override imports; resolve_configuration_async
  now accepts and forwards scoring_target_name so launch, estimate and
  dataset load keep the explicit scoring target with an exact selection.
- scenario_run_service: optional launch fields keep both
  scoring_target_name and example_selection.
- tests: patch resolve_target on the resolver class, as main's tests do,
  and cover scoring_target_name together with an exact selection.
LeakReplay (microsoft#3689, merged to main) only accepts LeakReplayDatasetConfiguration,
so an exact selection would be advertised and then rejected at launch.
…crosoft#2757)

The exact-selection compatibility check built its own factory map in the
resolver, letting a technique enum's catalog defaults replace registered
factories of the same name. TextAdaptive (and the base readiness view)
give registered factories precedence, so the check could validate a
different factory from the one that executes: a registered objective-only
crescendo accepted a conversation that initialization then skipped.

- Add Scenario._get_selected_technique_factories, which resolves the
  selection with _resolve_scenario_techniques and looks factories up in
  _get_readiness_factories, the view readiness and execution share.
- The resolver configures its introspection scenario with the run's
  scenario_params and resolved values, then checks those factories.
  Estimate and launch pass scenario_params through.
- Regression: real TextAdaptive with a registered objective-only crescendo
  override rejects a conversation plus objective as unsupported.
…crosoft#2757)

The technique compatibility check also ran during configured estimates and
rebuilt every selected group, so a stored simulated conversation was
reconstructed as SeedSimulatedConversation and read its legacy YAML
prompt paths. microsoft#2757 requires estimates not to load legacy templates.

- validate_technique_factories(inspect_only=True) builds the groups from
  stored inspection data. A simulated conversation becomes an unvalidated
  placeholder carrying its stored type, group, num_turns and sequence
  (model_construct), so no legacy path or default template is opened.
  The shared prepare/require_unchanged checks still run on that group.
- ScenarioService estimates pass inspect_exact_selection=True. Launch keeps
  full reconstruction and validation of the executable seeds.
- Regressions through ScenarioService.estimate_scenario_run_size_async with
  real RapidResponse, crescendo, an explicit scorer and a stored legacy
  simulated-conversation record: estimate count 1 with zero legacy-loader
  and template calls, and many_shot is still rejected as unsupported.
Preserve objective decomposition and technique flags when rebuilding stored seeds. Resolve exact groups through the shared dataset seam and reject population-changing overrides. Opt ASSERT out of REST exact selection until its selective tester and baseline policies can preserve the selection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25 hannahwestra25 self-assigned this Nov 16, 2026
Resolve conflicts with microsoft#4130 (trace-grounded ASSERT judging): keep main's v3 judge and VERSION 3, keep AssertPolicy.SUPPORTS_EXACT_EXAMPLE_SELECTION = False, and keep the exact-selection test on top of main's rewritten ASSERT tests.
@hannahwestra25 hannahwestra25 removed their assignment Nov 17, 2026
Resolve tests/unit/scenario/foundry/test_assert_policy.py by keeping
main's ASSERT metrics intent from microsoft#4131 plus the PR's exact selection
imports and tests. microsoft#4371 landed cleanly in tests/unit/backend/test_chat_scoring.py.
@richlundeen

Copy link
Copy Markdown
Contributor

Heads-up on an overlap with #4350, which changes how resume works.

#4350 stores each run's planned seed groups when the run is created (ScenarioPlanSeedGroupEntries), and resume rebuilds from those stored copies instead of re-reading datasets. The goal is that editing, refreshing, or deleting a dataset no longer breaks resume for a paused run. #3920's edit and delete routes make that an everyday action.

This PR goes the other way for exact selections. On resume, _prepare_run_async re-resolves the saved selection against the live datasets and returns 409 stale (or missing) if a selected row changed or was deleted; test_launch_request_round_trip_keeps_selection_and_resume_goes_stale pins that. With #4350 in place, the stored seed groups already hold exactly the rows the run planned, so that check would be redundant on resume, and it would bring back the failure #4350 removes, just for exact-selection runs.

Suggestion: keep the fingerprint check for preview, estimate, and launch, where it guards against running on content the user didn't see, and drop it from the resume path. Both PRs also touch memory_interface.py, scenario.py, and foundry/assert_policy.py, so whichever merges second will need to reconcile those.

@juliosuas

Copy link
Copy Markdown
Contributor Author

Thanks Richard Lundeen (@richlundeen), agreed. Once #4350 is in, resume should rebuild from the stored ScenarioPlanSeedGroupEntries and not re-resolve the selection against live datasets. Otherwise exact-selection runs get back the failure #4350 removes.

Plan:

I'll take care of reconciling memory_interface.py, scenario.py and foundry/assert_policy.py if this one merges second.

Resume now rebuilds from the seed groups microsoft#4350 stores with the run, so it
no longer re-resolves an exact selection against live datasets. Preview,
estimate and launch keep the fingerprint check.
@juliosuas

Copy link
Copy Markdown
Contributor Author

Richard Lundeen (@richlundeen) I merged main (with #4350) into this branch. Resume now relies on #4350's stored ScenarioPlanSeedGroupEntries:

  • On resume, the backend keeps the saved example_selection and still rejects a different one. It no longer resolves the selection against live datasets. The scenario gets an empty exact-selection config (ExactExampleDatasetConfiguration.for_resume()) and runs the stored seed groups. That config raises if anything tries to build groups from it.
  • The fingerprint check stays in preview, estimate and launch. An edited or deleted row still returns 409 there.
  • I replaced test_launch_request_round_trip_keeps_selection_and_resume_goes_stale with tests where a selected row is edited or deleted after launch. A new launch goes stale, and resume continues with the planned rows and the same run plan.

The only textual conflict was the scenario.py import, and both imports are kept. The other overlapping files merged cleanly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants