Skip to content

FEAT Add multipart seed dataset upload and deletion - #3920

Merged
Richard Lundeen (richlundeen) merged 17 commits into
microsoft:mainfrom
richlundeen:richlundeen-dataset-upload-and-delete
Nov 11, 2026
Merged

Richard Lundeen (richlundeen) merged 17 commits into
microsoft:mainfrom
richlundeen:richlundeen-dataset-upload-and-delete

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Manage stored seed datasets through the REST API: upload datasets with media, add/edit/delete individual seeds or batches, and delete whole datasets. Supports all three canonical seed types: prompt, objective, and simulated_conversation.

  • Extend the existing multipart endpoint: default create behavior stays unchanged; explicit update mode applies additions, edits, and delete_seed_ids atomically.
  • Supply editable seed representations and dataset version tokens through existing browsing routes. Updates require expected_version, preserve edited identities and omitted records, and validate the final affected groups before committing.
  • Accept applicable media as inert uploaded files, with scoped references for retaining existing media. Bounded ingestion, durable staging/recovery, and shared-reference checks protect uploads and cleanup; only API-owned, unshared files are removed.

Dataset endpoints

All paths include the /api prefix. Browsing uses logical examples/groups; mutations target the individual seed record IDs within them.

Method Endpoint Purpose Success
GET /api/datasets List registered and stored datasets; loaded_only=true limits results to stored populations. 200
POST /api/datasets Create a named dataset from a multipart JSON manifest and optional files. Default mode is create. 201
POST /api/datasets With mode: "update", add/edit seeds and remove delete_seed_ids atomically. Requires expected_version. 200
GET /api/datasets/seeds?selection_key=... Browse/filter/paginate examples. Add editable=true for submit-ready seeds and a dataset version. 200
GET /api/datasets/seeds/{example_id}?selection_key=... Read an example and its members; also supports editable=true. 200
DELETE /api/datasets?selection_key=... Delete the exact named stored population, preserving provider files and shared/external media. 200, or 202 while cleanup is pending
GET /api/datasets/deletions/{deletion_id} Read cleanup status and retry pending owned-media cleanup; also tracks mutation cleanup. 200

For example, submit this as the multipart manifest to add one seed, edit another, and delete a third in one transaction:

{
  "name": "my_dataset",
  "mode": "update",
  "expected_version": "<version returned with editable=true>",
  "seeds": [
    {"seed_type": "prompt", "data_type": "text", "value": "New prompt"},
    {"id": "<existing-record-uuid>", "seed_type": "prompt", "data_type": "text", "value": "Edited prompt"}
  ],
  "delete_seed_ids": ["<another-existing-record-uuid>"]
}

Entries without an ID are additions; entries with an ID replace that record's editable fields. Deletion-only requests use seeds: [] with nonempty delete_seed_ids; deleting the final seed removes the stored dataset population. Stale versions return 409; invalid targets or final groups reject the entire batch. Successful mutations return exact operation results and explicit cleanup status, including when physical cleanup is pending.

The backend README documents editable fields/defaults, group identities, multipart media, limits, and error responses. No frontend or export API is added.

Implement bounded multipart SeedPrompt imports, exact named dataset removal, and durable reference-safe media cleanup with SQLite and SQL Server migrations. Cover lifecycle failures, cancellation, limits, concurrency, restart recovery, and existing backend behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 4, 2026 08:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical lifecycle flaws can delete still-referenced media, persist rejected references, block startup recovery, and disrupt active uploads, alongside dataset identity and validation inconsistencies.

8 open issues
What changed in this PR

Adds backend dataset lifecycle APIs to PyRIT, allowing multipart seed uploads and named dataset deletion while retaining existing browsing and media endpoints.

Changes:

  • Adds bounded manifest/media uploads, validation, and sanitized API responses.
  • Introduces durable ownership and cleanup records, reference locking, restart recovery, and a database migration.
  • Adds lifecycle and storage tests, updates schema fixtures, and documents the API.
File Description
uv.lock Records the multipart dependency.
tests/​unit/​memory/​test_sqlite_fixtures.py Updates fixtures for migrated lifecycle tables.
tests/​unit/​memory/​test_score_participation_migration.py Allows the newer migration head.
tests/​unit/​memory/​test_dataset_lifecycle.py Tests reservation contention, references, and recovery.
tests/​unit/​memory/​test_dataset_lifecycle_migration.py Tests schema migration and SQL Server locking.
tests/​unit/​memory/​storage/​test_storage.py Tests streamed blob uploads and client cleanup.
tests/​unit/​backend/​test_dataset_lifecycle.py Tests upload, deletion, limits, rollback, and authentication.
pyrit/​memory/​storage/​storage.py Adds streaming disk/blob writes and offloads filesystem operations.
pyrit/​memory/​storage/​serializers.py Adds streamed media persistence.
pyrit/​memory/​memory_session.py Guards ORM and bulk media-reference writes.
pyrit/​memory/​memory_models.py Defines lifecycle, ownership, and lock tables.
pyrit/​memory/​memory_interface.py Exposes dataset lifecycle operations.
pyrit/​memory/​dataset_lifecycle.py Implements reservations, publication, deletion, and recovery.
pyrit/​memory/​alembic/​versions/​f8a2c6e0b419_add_dataset_lifecycle.py Creates durable lifecycle tables and indexes.
pyrit/​exceptions/​dataset_exception.py Defines sanitized dataset API failures.
pyrit/​backend/​services/​runtime_lifecycle.py Invokes cleanup recovery during startup.
pyrit/​backend/​services/​dataset_upload_service.py Parses, validates, and persists multipart imports.
pyrit/​backend/​services/​dataset_service.py Adds deletion and cleanup-status services.
pyrit/​backend/​routes/​datasets.py Exposes upload, deletion, and cleanup-status routes.
pyrit/​backend/​README.md Documents API usage, limits, and cleanup behavior.
pyrit/​backend/​models/​datasets.py Defines upload limits, manifests, and response models.
pyrit/​backend/​middleware/​request_size.py Applies dataset-specific request limits.
pyrit/​backend/​middleware/​error_handlers.py Returns dataset problem responses and cleanup locations.
pyproject.toml Declares the multipart parser dependency.

Review effort: Balanced

Comment thread pyrit/memory/dataset_lifecycle.py Resolved
Comment thread pyrit/memory/dataset_lifecycle.py Resolved
Comment thread pyrit/memory/dataset_lifecycle.py Resolved
Comment thread pyrit/memory/memory_session.py Resolved
Comment thread pyrit/memory/memory_session.py Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

Follow-up to this review.

Addressed all eight reported causes in 17a8436: early recovery, complete group validation, rejected-DML savepoint rollback, database-collated create-only identity, Unicode seed storage, native live-upload ownership, configured blob identity, and migrated test fixtures. Each inline has a separate reply with its regression evidence. The final affected regression batch passes 334 tests. SQL Server/Azure evidence remains offline DDL/mocked contracts; live cloud/code-page validation has not run. Independent review and the publisher's full pre-commit gate are still separate steps.

richlundeen

This comment was marked as duplicate.

@richlundeen
Richard Lundeen (richlundeen) marked this pull request as ready for review November 4, 2026 20:23
@richlundeen Richard Lundeen (richlundeen) changed the title [DRAFT] FEAT Add multipart seed dataset upload and deletion FEAT Add multipart seed dataset upload and deletion Nov 4, 2026
Preserve typed objectives, simulated conversation configuration and mixed attack groups while prevalidating untrusted nested prompts. Extend multipart media ownership and shared-reference protection to nested simulation prompts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
richlundeen

This comment was marked as duplicate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ccess

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

Preserve dataset mutations and portable browsing patterns, reconcile public imports and lifecycle fixtures, and join both Alembic heads.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use existing lazy package exports after main integration; preserve dataset behavior and assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

Every flush that wrote a message piece, seed or scorable content took the
global `DatasetLifecycleLock` (SQL Server `XLOCK, HOLDLOCK`) until commit,
so all conversation writes in a process ran one at a time. Because the
async engine runs each SQL call on an executor thread, a writer holding
the lock across an await could also wedge the backend: waiters filled
every thread and the holder never got one to commit (reproduced with 14
parallel attack creations at 8 threads / 15 connections).

Cleanup and deletion only inspect media-typed rows, so text-only writes
cannot add a dataset reference. Only seeds and rows carrying a media path
now take the lock and run reference validation; text-only writes still
start the SQLite write transaction first. On a Basic Azure SQL database,
40 parallel attack creations went from 38 s to 15 s and the 14-create
wedge no longer occurs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jbolor21 jbolor21 self-assigned this Nov 10, 2026
Comment thread pyrit/memory/dataset_lifecycle.py Resolved
Comment thread pyrit/memory/dataset_lifecycle.py Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve stored edit types, admit all simulation media slots, scope bulk reservation checks, and keep successful creates successful after native release errors.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 10, 2026
Main gained the seed decomposition migration (microsoft#4110), so this branch and main
each had an Alembic head and the merged schema failed to start. Add an empty
merge revision over both heads, and check that the merged-heads migration test
expects exactly one head rather than pinning its revision id, so a later
migration on main does not break it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen
Richard Lundeen (richlundeen) removed this pull request from the merge queue due to a manual request Nov 10, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to invalid changes in the merge commit Nov 10, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 10, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 11, 2026
Merged via the queue into microsoft:main with commit d6feb64 Nov 11, 2026
61 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-dataset-upload-and-delete branch November 11, 2026 00:39
Richard Lundeen (richlundeen) added a commit to richlundeen/PyRIT that referenced this pull request Nov 11, 2026
Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen) added a commit to richlundeen/PyRIT that referenced this pull request Nov 11, 2026
…(-4)

Brings in main 1dc1d5c (including dataset uploads, microsoft#3920) through -3 78f2477. Storage keeps both
list_files_async and main's write_stream_async/canonical_path; Azure write_stream_async now shares the
reference-counted container client so it can't close a client other operations are using.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants