Repository navigation
[FIX] Assign local YAML filename fallback to seeds so named queries find them - #3772
Conversation
…ind them A local YAML without dataset_name got a filename fallback on the provider/SeedDataset only; seed.dataset_name stayed None, so the memory insert persisted NULL and named queries/listings could not find the dataset. Assign the resolved name to unnamed seeds before returning, matching the per-seed assignment used by the generated and jailbreak loaders. Fixes microsoft#3740
Address review feedback: drop the stale issue reference in the comment/docstring, make both fixture seeds unnamed so DatasetConfiguration.prepare_async exercises the fallback path, and add a prepare_async regression. tests/unit/datasets: 5283 passed; ruff clean.
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the change that makes unnamed local YAML seeds discoverable through named memory queries.
No open issues 🎉
What changed in this PR
Fixes #3740 by propagating local YAML filename fallback names to unnamed seeds before persistence, making those seeds discoverable through named memory queries.
Changes:
- Assigns the provider’s resolved name to seeds without a dataset name, preserving existing names.
- Adds SQLite regression tests for named retrieval, dataset-name listing, scenario preparation, and explicit dataset names.
| File | Description |
|---|---|
tests/unit/datasets/test_local_dataset_loader.py |
Adds regression coverage for fallback names and explicit-name behavior. |
pyrit/datasets/seed_datasets/local/local_dataset_loader.py |
Assigns resolved dataset names to unnamed seeds before returning them. |
Review effort: Balanced
Add a keyword-only default_dataset_name to SeedDataset.from_yaml_file and load_seed_dataset_from_yaml. It is used only when the YAML declares neither dataset_name nor name, so _build_seeds applies its existing fill-only merge and seeds with their own dataset_name keep it. _LocalDatasetLoader passes its resolved name instead of re-stamping seeds after loading. Strengthen the explicit-name loader test with a separately named seed so an unconditional overwrite of per-seed names fails, and add model-level tests for the default (fills only unnamed seeds, declared dataset_name/name win, no-op when omitted). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
hannahwestra25
left a comment
There was a problem hiding this comment.
thanks for contributing :)
Resolve local dataset loader test imports using public package exports.
|
The only failing check is |
Brings in microsoft#3663 (reload a sending conversation after another one has loaded) and microsoft#3772. The ChatWindow conflict is resolved by keeping the extracted send workflow; microsoft#3663's markConversationLoaded() on the created conversation moves to the first-send branch of useChatSend, where the created attack is now routed. Adds a useChatSend test for it.
Fixes #3740.
Description
A local YAML dataset that omits
dataset_namegets a filename fallback at the provider/SeedDataset layer, but its seeds keepdataset_name=None. After the dataset is written to memory, named queries cannot find it:get_seeds_async(dataset_name="<stem>")returns 0 rows andget_seed_dataset_names_async()does not list the name, while the dataset itself appears in the provider list. An explicitly named YAML works.Root cause: the fallback name is decided in
LocalDatasetLoader.fetch_dataset_asyncafterSeedDataset._build_seedshas already run, so it never reachesseed.dataset_name, and the memory insert persists the NULL.The loader now assigns the resolved dataset name to seeds that do not have their own name before returning, matching the per-seed assignment already done by
assert_provider.pyandlocal/jailbreak_dataset.py. Explicit dataset-level and per-seed names are unchanged.Tests and Documentation
New regression cases in
tests/unit/datasets/test_local_dataset_loader.py:get_seed_dataset_names_async(); the explicit per-seed name is not overwritten;Evidence:
Also ran 289 related unit tests (seed datasets, local loader, dataset e2e contract, seed dataset provider) and 6 local integration tests;
ruff check,ruff format --check,ty check, andbuild_scripts/check_async_suffix.pypass on the changed files.Not run: the full unit suite, Windows/Python 3.11, and the full pre-commit set. No documentation change: this fixes an internal naming inconsistency, and the documented load/named-query contract is already what the fix makes true.