Skip to content

FEAT: Keep planned media for incomplete scenario runs - #4482

Open
Richard Lundeen (richlundeen) wants to merge 5 commits into
microsoft:mainfrom
richlundeen:keep-planned-media-for-incomplete-runs
Open

Richard Lundeen (richlundeen) wants to merge 5 commits into
microsoft:mainfrom
richlundeen:keep-planned-media-for-incomplete-runs

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Replaces #4365, which carries the same feature, with one commit on current main and a new migration revision. Previous: #4350 (seed 1, merged). This PR builds on it.

Description

Deleting a dataset no longer removes uploaded media files that a failed, cancelled, or paused scenario run planned, so that run can still resume after the delete. Seed 1 makes resume rebuild from the seed groups each run stored; this PR teaches dataset media cleanup to preserve the files those stored groups reference.

flowchart LR
  D["Dataset delete"] --> C["Media cleanup"]
  C --> K{"File still referenced?"}
  S["Seeds, messages, scores"] --> K
  P["Planned seed groups of runs<br/>not COMPLETED (new)"] --> K
  K -- yes --> R["Keep the file"]
  K -- no --> X["Delete the file"]
Loading
  • Adds planned-group media classification (planned_media_values, has_media, and its index) alongside its retention consumer. Migration cc714ae17d07 follows 3e55dd099e38, backfills existing stored groups in bounded pages, and preserves them on downgrade. On SQLite it keeps the trg_delete_scenario_plan_seed_groups trigger in place around the table rebuilds.
  • stored_media_keys counts media paths in flagged ScenarioPlanSeedGroupEntries rows whose run is not COMPLETED.
  • Planned seed group rows go through the same dataset reference guard as seeds and messages, under the lifecycle lock. A new run can't plan media already queued for deletion; it gets the existing "Media has been removed or is reserved for dataset cleanup" error.
  • Media of a completed run is deleted with its dataset as before.

A file kept because a run was incomplete stays on disk after that run completes. Nothing reclaims it, which is how other kept ("retained") dataset media behaves today.

#4365 used migration revision d2f4a6c8e0b3, which this PR doesn't include. A database upgraded on the #4365 branch fails the schema check here instead of skipping migrations. Downgrade it one revision, to c7d1e9a3b5f2, on that branch before switching; it then upgrades normally.

Deleting a dataset no longer removes media files that a paused or failed
scenario run planned and still needs to resume. Planned seed groups record
whether they reference media (has_media, indexed), and stored_media_keys
counts media from flagged groups whose run hasn't completed. COMPLETED runs
release their planned media as before.

Planned seed group writes join the dataset lifecycle flush and bulk-write
guards, so a run can't plan media that is already queued for deletion.

Migration cc714ae17d07 follows 3e55dd099e38, backfills has_media in bounded
pages, and keeps the SQLite trg_delete_scenario_plan_seed_groups trigger
around batch table rebuilds.

Replaces microsoft#4365 with a fresh migration identity on current main; follows microsoft#4350.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bulk writes revalidate every stored media reference. Planned references from
runs that aren't COMPLETED can name media deleted while the run was completed,
or before microsoft#4350's plans were counted, so they made every guarded bulk write
fail. Planned references still keep media during cleanup and are still checked
for bulk writes to planned seed groups.

Share the simulated conversation prompt field list between seed and planned
media reads, and drop a self-comparing test assertion.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The migration backfill can scale quadratically, and global media validation can incorrectly reject unrelated planned-group writes when historical media is missing.

2 open issues
What changed in this PR

Extends the stored-plan resume support from #4350 so dataset deletion preserves media needed by incomplete scenario runs.

Changes:

  • Classifies and indexes planned seed groups containing media, with a migration to backfill existing rows.
  • Includes incomplete runs’ planned media in cleanup retention and write-reservation checks.
  • Adds tests for media classification, lifecycle protection, and migration upgrade/downgrade behavior.
File Description
tests/​unit/​memory/​test_planned_seed_groups.py Tests media classification, including simulated conversations.
tests/​unit/​memory/​test_planned_seed_group_media_migration.py Tests backfill, downgrade preservation, and SQL Server SQL generation.
tests/​unit/​memory/​test_dataset_lifecycle.py Tests retention and cleanup-reservation guards for planned media.
pyrit/​memory/​memory_session.py Applies lifecycle guards to planned-group writes.
pyrit/​memory/​memory_models.py Adds planned-media extraction and an indexed media flag.
pyrit/​memory/​dataset_lifecycle.py Retains media referenced by incomplete runs’ stored plans.
pyrit/​memory/​alembic/​versions/​cc714ae17d07_flag_planned_seed_group_media.py Adds and backfills the media flag while preserving SQLite delete triggers.

Review effort: Balanced

Comment thread pyrit/memory/memory_session.py Resolved
Dataset cleanup holds the lifecycle lock while it reads scenario runs, so a
run that plans media now takes the lock before inserting its result row.
Bulk planned seed group writes check only the media references they add, so
an older plan naming deleted media no longer blocks unrelated plan inserts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants