Repository navigation
[BREAKING] FEAT: Resume scenario runs from their stored planned seed groups (seed 1) - #4350
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Restoration can re-render prompts and rebuild policy state from changed live inputs, breaking scenario replay, while default SQLite deletions leave planned inputs orphaned.
3 open issues
What changed in this PR
Moves PyRIT scenario resume to stored seed-group snapshots, preserving the original sampled inputs across dataset changes and duplicate occurrences (#4157). This supports the dataset editing and deletion workflows introduced by #3920.
Changes:
- Adds planned-seed storage and a migration, saving snapshots atomically with scenario results.
- Rebuilds resumed populations from snapshots while retaining configuration checks and scenario-specific setup hooks.
- Removes legacy resume support: older runs must finish before upgrading or be restarted.
| File | Description |
|---|---|
tests/unit/scenario/scenarios/technique/test_execute.py |
Updates resume configuration-mismatch expectations. |
tests/unit/scenario/scenarios/adaptive/test_text_adaptive.py |
Updates adaptive replay and legacy-run expectations. |
tests/unit/scenario/garak/test_latent_injection.py |
Checks replay of planned cases after input changes. |
tests/unit/scenario/core/test_scenario.py |
Covers snapshot replay, duplicates, and resume diagnostics. |
tests/unit/scenario/core/test_scenario_retry.py |
Removes tests for legacy objective replay. |
tests/unit/scenario/core/test_scenario_dataset_overrides.py |
Updates saved-selection and configuration checks. |
tests/unit/memory/test_scenario_plan_seed_groups_migration.py |
Adds migration and SQL Server schema checks. |
tests/unit/memory/test_planned_seed_groups.py |
Tests snapshot ordering, duplicates, and media detection. |
tests/unit/memory/test_dataset_lifecycle_migration.py |
Includes the new table in schema expectations. |
tests/unit/backend/test_scenario_resume.py |
Tracks the new atomic persistence API. |
pyrit/scenario/core/scenario.py |
Stores planned populations and restores them on resume. |
pyrit/memory/memory_models.py |
Defines snapshot rows and group reconstruction. |
pyrit/memory/memory_interface.py |
Adds transactional snapshot writes and ordered reads. |
pyrit/memory/alembic/versions/c7d1e9a3b5f2_add_scenario_plan_seed_groups.py |
Creates the planned-seed table and indexes. |
.github/instructions/scenarios.instructions.md |
Documents snapshot-based resume and override dependencies. |
Review effort: Balanced
|
Scope review at b184270: simplify before implementation review. Keep the ordered, run-owned seed snapshots and transactional persistence: they address a real resume workflow after dataset changes. Remove the unused snapshot media-classification scaffolding ( The recommendation was independently verified. The fix loop has stopped at its inclusion gate without source edits, integration, pushes, approval, or thread resolution. This is not a correctness review: the existing restoration, policy/naming, and cleanup feedback remains unassessed. Stored media paths do not preserve uploaded file bytes; that remains the explicitly deferred follow-up. |
Keep the replay layer independent of media-retention classification, which belongs to seed 2. Rebuilt locally from the exact reviewed candidate tree; original published and reviewed tips remain preserved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
b184270 to
0250130
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reconcile the migration heads, enforce snapshot ownership on SQLite, and restore prepared prompts and ASSERT names from the stored population. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…live reads on resume - Rebuild stored simulated conversations as instances so group validation keeps their sequence. - Psychosocial, AdversarialBenchmark, FigStep, WebInjection, SystemPromptExtraction and Exploitation resume from stored groups without re-running their resolvers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume now builds from the stored planned seed groups and no longer reruns an overriding seed resolver by default. The base _prepare_resume_state_async hook receives the stored population and does nothing, so the seven no-op overrides are gone. PromptInject, ApiKey, PackageHallucination and Execute override it to restore state their resolvers read from live data. LatentInjection rebuilds its trigger scorer from the stored groups' family/trigger pairs, so editing, adding or removing a trigger no longer refuses resume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The resolver docstring and the scenario instructions no longer say resume reruns an overriding resolver. ApiKey restores state on resume only when it builds the default credential scorer, so a custom scorer resumes without reading the corpus. Split the resolver resume test into one test per behavior. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
… PackageHallucination The resume hooks reran the whole resolver, so resume re-read population datasets that the stored groups already carry, and deleting one blocked resume. PromptInject now reads only its technique templates and rebuilds the goal scorer from goal_texts. ApiKey reads only the partial keys and placeholders its default scorer excludes. PackageHallucination reads only the selected registries, not the stub and task corpus. ScenarioRequiredData.load_async takes names for these subset reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the adaptive_scenario import conflict (each side had removed the code using the other's extra imports) and joins main's prompt generation migrations with a merge revision. Main's migration tests now check tables at their own revision, since this branch's planned-group migration is a sibling merged at head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e its state Resume no longer needs the population datasets, so listing every missing configured dataset could name a dataset resume doesn't read. A missing dataset read now raises MissingDatasetError with its name, and resume reports only that dataset; other errors are re-raised unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on mismatch Resume decided "deleted" from a second query of the dataset names in memory, so resuming with an added technique said its never-used registry was deleted. The stored identity lists the datasets the planned run read: a missing dataset outside it is now the configuration-mismatch error, and one inside it names that dataset as no longer in memory. ApiKey now names its exclusion datasets when an edit to them is what blocks resume. The adaptive branch for a missing run plan is removed, since base resume refuses such a result first, and three tests are tightened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onfiguration changed A scenario can read a dataset its configuration doesn't list, such as a PackageHallucination registry under a programmatic dataset_config, so a missing dataset outside the stored list isn't proof the selection changed. Resume now checks the stored configuration first and otherwise names the dataset as no longer in memory. ApiKey blames its exclusions only when the planned scorer excluded values, so a run planned with a custom scorer gets the generic mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume now rebuilds from the seed groups microsoft#4350 stores with the run, so it no longer re-resolves an exact selection against live datasets. Preview, estimate and launch keep the fingerprint check.


Description
Resuming a scenario run now rebuilds the seed groups the run planned from a copy stored when the run was created, so editing, refreshing, or deleting a dataset no longer breaks resume. Until #3920, datasets changed only through
RefreshDatasetsand SDK removal. Its REST edit and delete routes make changing a dataset under a paused run an everyday action, and resume then fails with "missing N planned seed group(s)" or "no seeds found in memory".flowchart LR subgraph new["New run"] D[(Datasets)] --> R[Resolve and sample population] R --> B[Build atomic attacks] B --> P["Store ScenarioResult, its plan,<br/>and its planned seed groups<br/>(one transaction)"] end subgraph resume["Resume"] L["Load the stored<br/>planned seed groups"] --> B2[Build atomic attacks] H["Opt-in resume hook:<br/>restore state the resolver sets up,<br/>from the stored groups when they carry it"] -.-> B2 B2 --> V[Check against the stored plan] --> C[Continue the run] end P -. scenario_result_id .-> L E["Dataset edit, refresh, or delete"] --> DScenarioPlanSeedGroupEntries(migrationc7d1e9a3b5f2): one row per planned seed group, holding the fullAttackSeedGroup, deleted with its scenario result.ScenarioResult(MemoryInterface.add_planned_scenario_result_async);get_planned_seed_groups_asyncreads it back._prepare_resume_state_async, which receives the stored groups and does nothing by default. LatentInjection rebuilds its trigger scorer from the stored groups; PromptInject, PackageHallucination, Execute, and ApiKey (with its default scorer) reread the live data they need. If a dataset that hook reads was deleted, resume names the dataset and the two ways to recover.objective_hashes,_apply_persisted_objectives, and rebuilding a plan for results without one._restore_planned_populationsis removed).AtomicAttack.keep_seed_groups_with_hashes, which only served the removed replay.Breaking: a run created before this change can't be resumed. Resume says it "was created by an older PyRIT version that didn't store the seeds it planned" and asks for a new run. Finish or restart paused runs before upgrading.
Rows hold the full group, about 1.8 KB per group on PyRIT's bundled datasets. Run history, analytics, and the GUI read the plan JSON, which is unchanged, not these rows.
Media files are not yet protected: deleting a dataset still removes its uploaded media even if an incomplete run planned it, so that run fails when it sends that seed. A follow-up PR keeps those files for incomplete runs.