Repository navigation
FEAT: Keep dataset media that an incomplete scenario run planned (seed 2) - #4365
Closed
Richard Lundeen (richlundeen) wants to merge 2 commits into
Closed
Richard Lundeen (richlundeen) wants to merge 2 commits into
Richard Lundeen (richlundeen) wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain to block approval.
No open issues 🎉
What changed in this PR
Builds on #4350’s stored scenario populations by preserving dataset media needed by incomplete runs. Dataset deletion now accounts for those stored references, allowing affected runs to resume without losing their planned media.
Changes:
- Retains media referenced by planned seed groups for runs that are not
COMPLETED. - Extends lifecycle locking and reference validation to prevent new plans from referencing media queued for deletion.
- Adds retention and deletion-guard tests. The supplied diff also includes #4350’s prerequisite storage, resume, and migration changes.
| File | Description |
|---|---|
tests/unit/scenario/scenarios/technique/test_execute.py |
Updates resume configuration-error expectations. |
tests/unit/scenario/scenarios/adaptive/test_text_adaptive.py |
Updates adaptive resume tests for stored populations. |
tests/unit/scenario/garak/test_latent_injection.py |
Verifies resume preserves planned cases after source changes. |
tests/unit/scenario/core/test_scenario.py |
Covers stored-population resume and legacy-run rejection. |
tests/unit/scenario/core/test_scenario_retry.py |
Removes obsolete objective-replay tests. |
tests/unit/scenario/core/test_scenario_dataset_overrides.py |
Aligns resume expectations with stored populations. |
tests/unit/scenario/core/test_atomic_attack.py |
Checks keep-set filter deprecation warnings. |
tests/unit/memory/test_scenario_plan_seed_groups_migration.py |
Tests planned-group migration and SQL Server schema compilation. |
tests/unit/memory/test_planned_seed_groups.py |
Tests population round-trips and media-path extraction. |
tests/unit/memory/test_dataset_lifecycle.py |
Tests incomplete-run media retention and deletion guards. |
tests/unit/memory/test_dataset_lifecycle_migration.py |
Includes planned-group storage in schema expectations. |
tests/unit/backend/test_scenario_resume.py |
Updates persistence interception to the planned-result API. |
pyrit/scenario/scenarios/adaptive/adaptive_scenario.py |
Rebuilds adaptive attacks from stored populations and choices. |
pyrit/scenario/core/scenario.py |
Stores planned populations and restores them on resume. |
pyrit/scenario/core/atomic_attack.py |
Deprecates the former resume keep-set filter. |
pyrit/memory/memory_session.py |
Applies lifecycle guards to planned seed-group writes. |
pyrit/memory/memory_models.py |
Defines planned-group storage and media extraction. |
pyrit/memory/memory_interface.py |
Adds transactional population storage and retrieval. |
pyrit/memory/dataset_lifecycle.py |
Counts incomplete-run planned media as live references. |
pyrit/memory/alembic/versions/c7d1e9a3b5f2_add_scenario_plan_seed_groups.py |
Creates indexed planned seed-group storage. |
.github/instructions/scenarios.instructions.md |
Documents stored-population resume behavior and resolver exceptions. |
Review effort: Balanced
Keep the replay layer independent of media-retention classification, which belongs to seed 2. Rebuilt locally from the exact reviewed candidate tree; original published and reviewed tips remain preserved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resume rebuilds a run from its stored planned seed groups, but dataset media cleanup didn't count those groups as references, so deleting a dataset could remove a file a paused run still needed. Planned seed groups now record whether they reference media, and a new migration adds that flag and its index and classifies existing rows from their stored seed groups. stored_media_keys reads media paths from flagged planned seed groups of runs that haven't completed. Planned seed group writes go through the same dataset reference guard as seeds and messages, so a new run can't plan media that is already queued for deletion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
force-pushed
the
keep-planned-media
branch
from
November 17, 2026 22:31
985d8a6 to
4cfc02e
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previous: #4350 (seed 1). This PR builds on it; until seed 1 merges, review the retention layer above its head.
Description
Deleting a dataset no longer removes uploaded media files that a failed, cancelled, or paused scenario run planned, so that run can still resume after the delete. Seed 1 makes resume rebuild from the seed groups each run stored; this PR teaches dataset media cleanup to preserve the files those stored groups reference.
flowchart LR D["Dataset delete"] --> C["Media cleanup"] C --> K{"File still referenced?"} S["Seeds, messages, scores"] --> K P["Planned seed groups of runs<br/>not COMPLETED (new)"] --> K K -- yes --> R["Keep the file"] K -- no --> X["Delete the file"]planned_media_values,has_media, and its index) alongside its retention consumer. A separate migration backfills existing stored groups and preserves them on downgrade.stored_media_keyscounts media paths in flaggedScenarioPlanSeedGroupEntriesrows whose run is notCOMPLETED.A file kept because a run was incomplete stays on disk after that run completes. Nothing reclaims it, which is how other kept ("retained") dataset media behaves today.