Skip to content

FEAT: Keep dataset media that an incomplete scenario run planned (seed 2) - #4365

Closed
Richard Lundeen (richlundeen) wants to merge 2 commits into
microsoft:mainfrom
richlundeen:keep-planned-media
Closed

Richard Lundeen (richlundeen) wants to merge 2 commits into
microsoft:mainfrom
richlundeen:keep-planned-media

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Previous: #4350 (seed 1). This PR builds on it; until seed 1 merges, review the retention layer above its head.

Description

Deleting a dataset no longer removes uploaded media files that a failed, cancelled, or paused scenario run planned, so that run can still resume after the delete. Seed 1 makes resume rebuild from the seed groups each run stored; this PR teaches dataset media cleanup to preserve the files those stored groups reference.

flowchart LR
  D["Dataset delete"] --> C["Media cleanup"]
  C --> K{"File still referenced?"}
  S["Seeds, messages, scores"] --> K
  P["Planned seed groups of runs<br/>not COMPLETED (new)"] --> K
  K -- yes --> R["Keep the file"]
  K -- no --> X["Delete the file"]
Loading
  • Adds planned-group media classification (planned_media_values, has_media, and its index) alongside its retention consumer. A separate migration backfills existing stored groups and preserves them on downgrade.
  • stored_media_keys counts media paths in flagged ScenarioPlanSeedGroupEntries rows whose run is not COMPLETED.
  • Planned seed group rows go through the same dataset reference guard as seeds and messages, under the lifecycle lock. A new run can't plan media already queued for deletion; it gets the existing "Media has been removed or is reserved for dataset cleanup" error.
  • Media of a completed run is deleted with its dataset as before.

A file kept because a run was incomplete stays on disk after that run completes. Nothing reclaims it, which is how other kept ("retained") dataset media behaves today.

Copilot AI balanced review requested due to automatic review settings November 17, 2026 17:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain to block approval.

No open issues 🎉

What changed in this PR

Builds on #4350’s stored scenario populations by preserving dataset media needed by incomplete runs. Dataset deletion now accounts for those stored references, allowing affected runs to resume without losing their planned media.

Changes:

  • Retains media referenced by planned seed groups for runs that are not COMPLETED.
  • Extends lifecycle locking and reference validation to prevent new plans from referencing media queued for deletion.
  • Adds retention and deletion-guard tests. The supplied diff also includes #4350’s prerequisite storage, resume, and migration changes.
File Description
tests/​unit/​scenario/​scenarios/​technique/​test_execute.py Updates resume configuration-error expectations.
tests/​unit/​scenario/​scenarios/​adaptive/​test_text_adaptive.py Updates adaptive resume tests for stored populations.
tests/​unit/​scenario/​garak/​test_latent_injection.py Verifies resume preserves planned cases after source changes.
tests/​unit/​scenario/​core/​test_scenario.py Covers stored-population resume and legacy-run rejection.
tests/​unit/​scenario/​core/​test_scenario_retry.py Removes obsolete objective-replay tests.
tests/​unit/​scenario/​core/​test_scenario_dataset_overrides.py Aligns resume expectations with stored populations.
tests/​unit/​scenario/​core/​test_atomic_attack.py Checks keep-set filter deprecation warnings.
tests/​unit/​memory/​test_scenario_plan_seed_groups_migration.py Tests planned-group migration and SQL Server schema compilation.
tests/​unit/​memory/​test_planned_seed_groups.py Tests population round-trips and media-path extraction.
tests/​unit/​memory/​test_dataset_lifecycle.py Tests incomplete-run media retention and deletion guards.
tests/​unit/​memory/​test_dataset_lifecycle_migration.py Includes planned-group storage in schema expectations.
tests/​unit/​backend/​test_scenario_resume.py Updates persistence interception to the planned-result API.
pyrit/​scenario/​scenarios/​adaptive/​adaptive_scenario.py Rebuilds adaptive attacks from stored populations and choices.
pyrit/​scenario/​core/​scenario.py Stores planned populations and restores them on resume.
pyrit/​scenario/​core/​atomic_attack.py Deprecates the former resume keep-set filter.
pyrit/​memory/​memory_session.py Applies lifecycle guards to planned seed-group writes.
pyrit/​memory/​memory_models.py Defines planned-group storage and media extraction.
pyrit/​memory/​memory_interface.py Adds transactional population storage and retrieval.
pyrit/​memory/​dataset_lifecycle.py Counts incomplete-run planned media as live references.
pyrit/​memory/​alembic/​versions/​c7d1e9a3b5f2_add_scenario_plan_seed_groups.py Creates indexed planned seed-group storage.
.github/​instructions/​scenarios.instructions.md Documents stored-population resume behavior and resolver exceptions.

Review effort: Balanced

Keep the replay layer independent of media-retention classification, which belongs to seed 2. Rebuilt locally from the exact reviewed candidate tree; original published and reviewed tips remain preserved.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resume rebuilds a run from its stored planned seed groups, but dataset media
cleanup didn't count those groups as references, so deleting a dataset could
remove a file a paused run still needed. Planned seed groups now record whether
they reference media, and a new migration adds that flag and its index and
classifies existing rows from their stored seed groups. stored_media_keys reads
media paths from flagged planned seed groups of runs that haven't completed.
Planned seed group writes go through the same dataset reference guard as seeds
and messages, so a new run can't plan media that is already queued for deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

Superseded by #4482, which carries this change rebuilt on current main (after #4350) with a fresh migration revision (cc714ae17d07) instead of editing the already-committed d2f4a6c8e0b3. Closing this PR in favor of it; please continue review there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants