Repository navigation
FEAT Add multipart seed dataset upload and deletion - #3920
Richard Lundeen (richlundeen) merged 17 commits into
Conversation
Implement bounded multipart SeedPrompt imports, exact named dataset removal, and durable reference-safe media cleanup with SQLite and SQL Server migrations. Cover lifecycle failures, cancellation, limits, concurrency, restart recovery, and existing backend behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical lifecycle flaws can delete still-referenced media, persist rejected references, block startup recovery, and disrupt active uploads, alongside dataset identity and validation inconsistencies.
8 open issues
Validate remapped seed groups before reserving or writing media · New Run upload recovery before configured initializers · New Use a Unicode column for seed dataset names · New Canonicalize media references using the storage backend’s identity · New Reject dataset names that collide under SQL Server collation · New Reclaim staging uploads only after confirming their owner is inactive · New Roll back the transaction when reference validation fails · New Update manual-schema test fixtures to create the lifecycle lock row · New
What changed in this PR
Adds backend dataset lifecycle APIs to PyRIT, allowing multipart seed uploads and named dataset deletion while retaining existing browsing and media endpoints.
Changes:
- Adds bounded manifest/media uploads, validation, and sanitized API responses.
- Introduces durable ownership and cleanup records, reference locking, restart recovery, and a database migration.
- Adds lifecycle and storage tests, updates schema fixtures, and documents the API.
| File | Description |
|---|---|
uv.lock |
Records the multipart dependency. |
tests/unit/memory/test_sqlite_fixtures.py |
Updates fixtures for migrated lifecycle tables. |
tests/unit/memory/test_score_participation_migration.py |
Allows the newer migration head. |
tests/unit/memory/test_dataset_lifecycle.py |
Tests reservation contention, references, and recovery. |
tests/unit/memory/test_dataset_lifecycle_migration.py |
Tests schema migration and SQL Server locking. |
tests/unit/memory/storage/test_storage.py |
Tests streamed blob uploads and client cleanup. |
tests/unit/backend/test_dataset_lifecycle.py |
Tests upload, deletion, limits, rollback, and authentication. |
pyrit/memory/storage/storage.py |
Adds streaming disk/blob writes and offloads filesystem operations. |
pyrit/memory/storage/serializers.py |
Adds streamed media persistence. |
pyrit/memory/memory_session.py |
Guards ORM and bulk media-reference writes. |
pyrit/memory/memory_models.py |
Defines lifecycle, ownership, and lock tables. |
pyrit/memory/memory_interface.py |
Exposes dataset lifecycle operations. |
pyrit/memory/dataset_lifecycle.py |
Implements reservations, publication, deletion, and recovery. |
pyrit/memory/alembic/versions/f8a2c6e0b419_add_dataset_lifecycle.py |
Creates durable lifecycle tables and indexes. |
pyrit/exceptions/dataset_exception.py |
Defines sanitized dataset API failures. |
pyrit/backend/services/runtime_lifecycle.py |
Invokes cleanup recovery during startup. |
pyrit/backend/services/dataset_upload_service.py |
Parses, validates, and persists multipart imports. |
pyrit/backend/services/dataset_service.py |
Adds deletion and cleanup-status services. |
pyrit/backend/routes/datasets.py |
Exposes upload, deletion, and cleanup-status routes. |
pyrit/backend/README.md |
Documents API usage, limits, and cleanup behavior. |
pyrit/backend/models/datasets.py |
Defines upload limits, manifests, and response models. |
pyrit/backend/middleware/request_size.py |
Applies dataset-specific request limits. |
pyrit/backend/middleware/error_handlers.py |
Returns dataset problem responses and cleanup locations. |
pyproject.toml |
Declares the multipart parser dependency. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Follow-up to this review. Addressed all eight reported causes in 17a8436: early recovery, complete group validation, rejected-DML savepoint rollback, database-collated create-only identity, Unicode seed storage, native live-upload ownership, configured blob identity, and migrated test fixtures. Each inline has a separate reply with its regression evidence. The final affected regression batch passes 334 tests. SQL Server/Azure evidence remains offline DDL/mocked contracts; live cloud/code-page validation has not run. Independent review and the publisher's full pre-commit gate are still separate steps. |
Preserve typed objectives, simulated conversation configuration and mixed attack groups while prevalidating untrusted nested prompts. Extend multipart media ownership and shared-reference protection to nested simulation prompts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ccess Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.
Preserve dataset mutations and portable browsing patterns, reconcile public imports and lifecycle fixtures, and join both Alembic heads. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use existing lazy package exports after main integration; preserve dataset behavior and assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.
Every flush that wrote a message piece, seed or scorable content took the global `DatasetLifecycleLock` (SQL Server `XLOCK, HOLDLOCK`) until commit, so all conversation writes in a process ran one at a time. Because the async engine runs each SQL call on an executor thread, a writer holding the lock across an await could also wedge the backend: waiters filled every thread and the holder never got one to commit (reproduced with 14 parallel attack creations at 8 threads / 15 connections). Cleanup and deletion only inspect media-typed rows, so text-only writes cannot add a dataset reference. Only seeds and rows carrying a media path now take the lock and run reference validation; text-only writes still start the SQLite write transaction first. On a Basic Azure SQL database, 40 parallel attack creations went from 38 s to 15 s and the 14-create wedge no longer occurs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve stored edit types, admit all simulation media slots, scope bulk reservation checks, and keep successful creates successful after native release errors. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Main gained the seed decomposition migration (microsoft#4110), so this branch and main each had an Alembic head and the merged schema failed to start. Add an empty merge revision over both heads, and check that the merged-heads migration test expects exactly one head rather than pinning its revision id, so a later migration on main does not break it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…(-4) Brings in main 1dc1d5c (including dataset uploads, microsoft#3920) through -3 78f2477. Storage keeps both list_files_async and main's write_stream_async/canonical_path; Azure write_stream_async now shares the reference-counted container client so it can't close a client other operations are using. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


Description
Manage stored seed datasets through the REST API: upload datasets with media, add/edit/delete individual seeds or batches, and delete whole datasets. Supports all three canonical seed types:
prompt,objective, andsimulated_conversation.createbehavior stays unchanged; explicitupdatemode applies additions, edits, anddelete_seed_idsatomically.expected_version, preserve edited identities and omitted records, and validate the final affected groups before committing.Dataset endpoints
All paths include the
/apiprefix. Browsing uses logical examples/groups; mutations target the individual seed record IDs within them.GET/api/datasetsloaded_only=truelimits results to stored populations.200POST/api/datasetsmanifestand optional files. Default mode iscreate.201POST/api/datasetsmode: "update", add/editseedsand removedelete_seed_idsatomically. Requiresexpected_version.200GET/api/datasets/seeds?selection_key=...editable=truefor submit-ready seeds and a dataset version.200GET/api/datasets/seeds/{example_id}?selection_key=...editable=true.200DELETE/api/datasets?selection_key=...200, or202while cleanup is pendingGET/api/datasets/deletions/{deletion_id}200For example, submit this as the multipart
manifestto add one seed, edit another, and delete a third in one transaction:{ "name": "my_dataset", "mode": "update", "expected_version": "<version returned with editable=true>", "seeds": [ {"seed_type": "prompt", "data_type": "text", "value": "New prompt"}, {"id": "<existing-record-uuid>", "seed_type": "prompt", "data_type": "text", "value": "Edited prompt"} ], "delete_seed_ids": ["<another-existing-record-uuid>"] }Entries without an ID are additions; entries with an ID replace that record's editable fields. Deletion-only requests use
seeds: []with nonemptydelete_seed_ids; deleting the final seed removes the stored dataset population. Stale versions return409; invalid targets or final groups reject the entire batch. Successful mutations return exact operation results and explicit cleanup status, including when physical cleanup is pending.The backend README documents editable fields/defaults, group identities, multipart media, limits, and error responses. No frontend or export API is added.