Skip to content

[BREAKING] FEAT: Resume scenario runs from their stored planned seed groups (seed 1) - #4350

Merged
Richard Lundeen (richlundeen) merged 14 commits into
microsoft:mainfrom
richlundeen:resume-from-planned-seeds
Nov 20, 2026
Merged

Richard Lundeen (richlundeen) merged 14 commits into
microsoft:mainfrom
richlundeen:resume-from-planned-seeds

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Resuming a scenario run now rebuilds the seed groups the run planned from a copy stored when the run was created, so editing, refreshing, or deleting a dataset no longer breaks resume. Until #3920, datasets changed only through RefreshDatasets and SDK removal. Its REST edit and delete routes make changing a dataset under a paused run an everyday action, and resume then fails with "missing N planned seed group(s)" or "no seeds found in memory".

flowchart LR
  subgraph new["New run"]
    D[(Datasets)] --> R[Resolve and sample population]
    R --> B[Build atomic attacks]
    B --> P["Store ScenarioResult, its plan,<br/>and its planned seed groups<br/>(one transaction)"]
  end
  subgraph resume["Resume"]
    L["Load the stored<br/>planned seed groups"] --> B2[Build atomic attacks]
    H["Opt-in resume hook:<br/>restore state the resolver sets up,<br/>from the stored groups when they carry it"] -.-> B2
    B2 --> V[Check against the stored plan] --> C[Continue the run]
  end
  P -. scenario_result_id .-> L
  E["Dataset edit, refresh, or delete"] --> D
Loading
  • Adds ScenarioPlanSeedGroupEntries (migration c7d1e9a3b5f2): one row per planned seed group, holding the full AttackSeedGroup, deleted with its scenario result.
  • New runs store their sampled population in the same transaction as the ScenarioResult (MemoryInterface.add_planned_scenario_result_async); get_planned_seed_groups_async reads it back.
  • Resume no longer prepares, resolves, or resamples population datasets. A scenario whose resolver sets state its atomic attacks need restores it in _prepare_resume_state_async, which receives the stored groups and does nothing by default. LatentInjection rebuilds its trigger scorer from the stored groups; PromptInject, PackageHallucination, Execute, and ApiKey (with its default scorer) reread the live data they need. If a dataset that hook reads was deleted, resume names the dataset and the two ways to recover.
  • Fixes BUG: Resume fails for good when a dataset has an exact duplicate row and the first run sampled only one copy #4157: the stored population keeps exactly the copies the first run sampled.
  • Removes the legacy replay that re-resolved datasets: objective_hashes, _apply_persisted_objectives, and rebuilding a plan for results without one.
  • Simplifies AdaptiveScenario's resume: it receives the exact planned population, so it builds and names atomics like a new run and reads only its saved technique choices from the plan (_restore_planned_populations is removed).
  • Deprecates AtomicAttack.keep_seed_groups_with_hashes, which only served the removed replay.

Breaking: a run created before this change can't be resumed. Resume says it "was created by an older PyRIT version that didn't store the seeds it planned" and asks for a new run. Finish or restart paused runs before upgrading.

Rows hold the full group, about 1.8 KB per group on PyRIT's bundled datasets. Run history, analytics, and the GUI read the plan JSON, which is unchanged, not these rows.

Media files are not yet protected: deleting a dataset still removes its uploaded media even if an incomplete run planned it, so that run fails when it sends that seed. A follow-up PR keeps those files for incomplete runs.

Copilot AI balanced review requested due to automatic review settings November 17, 2026 03:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Restoration can re-render prompts and rebuild policy state from changed live inputs, breaking scenario replay, while default SQLite deletions leave planned inputs orphaned.

3 open issues
What changed in this PR

Moves PyRIT scenario resume to stored seed-group snapshots, preserving the original sampled inputs across dataset changes and duplicate occurrences (#4157). This supports the dataset editing and deletion workflows introduced by #3920.

Changes:

  • Adds planned-seed storage and a migration, saving snapshots atomically with scenario results.
  • Rebuilds resumed populations from snapshots while retaining configuration checks and scenario-specific setup hooks.
  • Removes legacy resume support: older runs must finish before upgrading or be restarted.
File Description
tests/​unit/​scenario/​scenarios/​technique/​test_execute.py Updates resume configuration-mismatch expectations.
tests/​unit/​scenario/​scenarios/​adaptive/​test_text_adaptive.py Updates adaptive replay and legacy-run expectations.
tests/​unit/​scenario/​garak/​test_latent_injection.py Checks replay of planned cases after input changes.
tests/​unit/​scenario/​core/​test_scenario.py Covers snapshot replay, duplicates, and resume diagnostics.
tests/​unit/​scenario/​core/​test_scenario_retry.py Removes tests for legacy objective replay.
tests/​unit/​scenario/​core/​test_scenario_dataset_overrides.py Updates saved-selection and configuration checks.
tests/​unit/​memory/​test_scenario_plan_seed_groups_migration.py Adds migration and SQL Server schema checks.
tests/​unit/​memory/​test_planned_seed_groups.py Tests snapshot ordering, duplicates, and media detection.
tests/​unit/​memory/​test_dataset_lifecycle_migration.py Includes the new table in schema expectations.
tests/​unit/​backend/​test_scenario_resume.py Tracks the new atomic persistence API.
pyrit/​scenario/​core/​scenario.py Stores planned populations and restores them on resume.
pyrit/​memory/​memory_models.py Defines snapshot rows and group reconstruction.
pyrit/​memory/​memory_interface.py Adds transactional snapshot writes and ordered reads.
pyrit/​memory/​alembic/​versions/​c7d1e9a3b5f2_add_scenario_plan_seed_groups.py Creates the planned-seed table and indexes.
.github/​instructions/​scenarios.instructions.md Documents snapshot-based resume and override dependencies.

Review effort: Balanced

Comment thread pyrit/memory/memory_models.py Resolved
Comment thread pyrit/scenario/core/scenario.py Resolved
Comment thread pyrit/memory/memory_models.py Resolved
@richlundeen Richard Lundeen (richlundeen) changed the title [BREAKING] FEAT: Resume scenario runs from their stored planned seed groups FEAT: Resume scenario runs from their stored planned seed groups Nov 17, 2026
@richlundeen Richard Lundeen (richlundeen) changed the title FEAT: Resume scenario runs from their stored planned seed groups [BREAKING] FEAT: Resume scenario runs from their stored planned seed groups (seed 1) Nov 17, 2026
@richlundeen

Copy link
Copy Markdown
Contributor Author

Scope review at b184270: simplify before implementation review.

Keep the ordered, run-owned seed snapshots and transactional persistence: they address a real resume workflow after dataset changes. Remove the unused snapshot media-classification scaffolding (planned_media_values, its simulated-conversation field traversal, and ScenarioPlanSeedGroupEntry.has_media with its migration column/index and classification-only tests). No production decision or query filter consumes that classification at this head; snapshot restoration reads the stored group JSON in position order without it. Removing it preserves the currently delivered replay behavior, including media path values. Add classification with its actual retention consumer, rather than reserve schema for the follow-up. This follows the framework's "Don't over-engineer" and end-to-end unit-of-value guidance.

The recommendation was independently verified. The fix loop has stopped at its inclusion gate without source edits, integration, pushes, approval, or thread resolution. This is not a correctness review: the existing restoration, policy/naming, and cleanup feedback remains unassessed. Stored media paths do not preserve uploaded file bytes; that remains the explicitly deferred follow-up.

Keep the replay layer independent of media-retention classification, which belongs to seed 2. Rebuilt locally from the exact reviewed candidate tree; original published and reviewed tips remain preserved.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reconcile the migration heads, enforce snapshot ownership on SQLite, and restore prepared prompts and ASSERT names from the stored population.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

Comment thread pyrit/memory/memory_models.py Resolved
Comment thread pyrit/scenario/core/scenario.py Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…live reads on resume

- Rebuild stored simulated conversations as instances so group validation keeps their sequence.
- Psychosocial, AdversarialBenchmark, FigStep, WebInjection, SystemPromptExtraction and Exploitation resume from stored groups without re-running their resolvers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

Comment thread pyrit/scenario/core/scenario.py Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume now builds from the stored planned seed groups and no longer reruns
an overriding seed resolver by default. The base _prepare_resume_state_async
hook receives the stored population and does nothing, so the seven no-op
overrides are gone. PromptInject, ApiKey, PackageHallucination and Execute
override it to restore state their resolvers read from live data.
LatentInjection rebuilds its trigger scorer from the stored groups'
family/trigger pairs, so editing, adding or removing a trigger no longer
refuses resume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The resolver docstring and the scenario instructions no longer say resume
reruns an overriding resolver. ApiKey restores state on resume only when it
builds the default credential scorer, so a custom scorer resumes without
reading the corpus. Split the resolver resume test into one test per
behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit

… PackageHallucination

The resume hooks reran the whole resolver, so resume re-read population
datasets that the stored groups already carry, and deleting one blocked
resume. PromptInject now reads only its technique templates and rebuilds
the goal scorer from goal_texts. ApiKey reads only the partial keys and
placeholders its default scorer excludes. PackageHallucination reads only
the selected registries, not the stub and task corpus.
ScenarioRequiredData.load_async takes names for these subset reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the adaptive_scenario import conflict (each side had removed
the code using the other's extra imports) and joins main's prompt
generation migrations with a merge revision. Main's migration tests now
check tables at their own revision, since this branch's planned-group
migration is a sibling merged at head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e its state

Resume no longer needs the population datasets, so listing every missing
configured dataset could name a dataset resume doesn't read. A missing
dataset read now raises MissingDatasetError with its name, and resume
reports only that dataset; other errors are re-raised unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on mismatch

Resume decided "deleted" from a second query of the dataset names in
memory, so resuming with an added technique said its never-used registry
was deleted. The stored identity lists the datasets the planned run read:
a missing dataset outside it is now the configuration-mismatch error, and
one inside it names that dataset as no longer in memory.

ApiKey now names its exclusion datasets when an edit to them is what
blocks resume. The adaptive branch for a missing run plan is removed,
since base resume refuses such a result first, and three tests are
tightened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onfiguration changed

A scenario can read a dataset its configuration doesn't list, such as a
PackageHallucination registry under a programmatic dataset_config, so a
missing dataset outside the stored list isn't proof the selection
changed. Resume now checks the stored configuration first and otherwise
names the dataset as no longer in memory. ApiKey blames its exclusions
only when the planned scorer excluded values, so a run planned with a
custom scorer gets the generic mismatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 20, 2026
Merged via the queue into microsoft:main with commit 1c0f4a6 Nov 20, 2026
65 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the resume-from-planned-seeds branch November 20, 2026 20:23
Julio César Suástegui (juliosuas) added a commit to juliosuas/PyRIT that referenced this pull request Nov 21, 2026
Resume now rebuilds from the seed groups microsoft#4350 stores with the run, so it
no longer re-resolves an exact selection against live datasets. Preview,
estimate and launch keep the fingerprint check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: Resume fails for good when a dataset has an exact duplicate row and the first run sampled only one copy

3 participants