Repository navigation
FEAT: Keep planned media for incomplete scenario runs - #4482
Open
Richard Lundeen (richlundeen) wants to merge 5 commits into
Open
Richard Lundeen (richlundeen) wants to merge 5 commits into
Richard Lundeen (richlundeen) wants to merge 5 commits into
Conversation
Deleting a dataset no longer removes media files that a paused or failed scenario run planned and still needs to resume. Planned seed groups record whether they reference media (has_media, indexed), and stored_media_keys counts media from flagged groups whose run hasn't completed. COMPLETED runs release their planned media as before. Planned seed group writes join the dataset lifecycle flush and bulk-write guards, so a run can't plan media that is already queued for deletion. Migration cc714ae17d07 follows 3e55dd099e38, backfills has_media in bounded pages, and keeps the SQLite trg_delete_scenario_plan_seed_groups trigger around batch table rebuilds. Replaces microsoft#4365 with a fresh migration identity on current main; follows microsoft#4350. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bulk writes revalidate every stored media reference. Planned references from runs that aren't COMPLETED can name media deleted while the run was completed, or before microsoft#4350's plans were counted, so they made every guarded bulk write fail. Planned references still keep media during cleanup and are still checked for bulk writes to planned seed groups. Share the simulated conversation prompt field list between seed and planned media reads, and drop a self-comparing test assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The migration backfill can scale quadratically, and global media validation can incorrectly reject unrelated planned-group writes when historical media is missing.
2 open issues
What changed in this PR
Extends the stored-plan resume support from #4350 so dataset deletion preserves media needed by incomplete scenario runs.
Changes:
- Classifies and indexes planned seed groups containing media, with a migration to backfill existing rows.
- Includes incomplete runs’ planned media in cleanup retention and write-reservation checks.
- Adds tests for media classification, lifecycle protection, and migration upgrade/downgrade behavior.
| File | Description |
|---|---|
tests/unit/memory/test_planned_seed_groups.py |
Tests media classification, including simulated conversations. |
tests/unit/memory/test_planned_seed_group_media_migration.py |
Tests backfill, downgrade preservation, and SQL Server SQL generation. |
tests/unit/memory/test_dataset_lifecycle.py |
Tests retention and cleanup-reservation guards for planned media. |
pyrit/memory/memory_session.py |
Applies lifecycle guards to planned-group writes. |
pyrit/memory/memory_models.py |
Adds planned-media extraction and an indexed media flag. |
pyrit/memory/dataset_lifecycle.py |
Retains media referenced by incomplete runs’ stored plans. |
pyrit/memory/alembic/versions/cc714ae17d07_flag_planned_seed_group_media.py |
Adds and backfills the media flag while preserving SQLite delete triggers. |
Review effort: Balanced
Dataset cleanup holds the lifecycle lock while it reads scenario runs, so a run that plans media now takes the lock before inserting its result row. Bulk planned seed group writes check only the media references they add, so an older plan naming deleted media no longer blocks unrelated plan inserts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
Contributor
Author
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Replaces #4365, which carries the same feature, with one commit on current
mainand a new migration revision. Previous: #4350 (seed 1, merged). This PR builds on it.Description
Deleting a dataset no longer removes uploaded media files that a failed, cancelled, or paused scenario run planned, so that run can still resume after the delete. Seed 1 makes resume rebuild from the seed groups each run stored; this PR teaches dataset media cleanup to preserve the files those stored groups reference.
flowchart LR D["Dataset delete"] --> C["Media cleanup"] C --> K{"File still referenced?"} S["Seeds, messages, scores"] --> K P["Planned seed groups of runs<br/>not COMPLETED (new)"] --> K K -- yes --> R["Keep the file"] K -- no --> X["Delete the file"]planned_media_values,has_media, and its index) alongside its retention consumer. Migrationcc714ae17d07follows3e55dd099e38, backfills existing stored groups in bounded pages, and preserves them on downgrade. On SQLite it keeps thetrg_delete_scenario_plan_seed_groupstrigger in place around the table rebuilds.stored_media_keyscounts media paths in flaggedScenarioPlanSeedGroupEntriesrows whose run is notCOMPLETED.A file kept because a run was incomplete stays on disk after that run completes. Nothing reclaims it, which is how other kept ("retained") dataset media behaves today.
#4365 used migration revision
d2f4a6c8e0b3, which this PR doesn't include. A database upgraded on the #4365 branch fails the schema check here instead of skipping migrations. Downgrade it one revision, toc7d1e9a3b5f2, on that branch before switching; it then upgrades normally.