Skip to content

[FIX] Assign local YAML filename fallback to seeds so named queries find them - #3772

Merged
hannahwestra25 merged 6 commits into
microsoft:mainfrom
feiiiiii5:fix/local-yaml-dataset-name
Nov 10, 2026
Merged

hannahwestra25 merged 6 commits into
microsoft:mainfrom
feiiiiii5:fix/local-yaml-dataset-name

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

Fixes #3740.

Description

A local YAML dataset that omits dataset_name gets a filename fallback at the provider/SeedDataset layer, but its seeds keep dataset_name=None. After the dataset is written to memory, named queries cannot find it: get_seeds_async(dataset_name="<stem>") returns 0 rows and get_seed_dataset_names_async() does not list the name, while the dataset itself appears in the provider list. An explicitly named YAML works.

Root cause: the fallback name is decided in LocalDatasetLoader.fetch_dataset_async after SeedDataset._build_seeds has already run, so it never reaches seed.dataset_name, and the memory insert persists the NULL.

The loader now assigns the resolved dataset name to seeds that do not have their own name before returning, matching the per-seed assignment already done by assert_provider.py and local/jailbreak_dataset.py. Explicit dataset-level and per-seed names are unchanged.

Tests and Documentation

New regression cases in tests/unit/datasets/test_local_dataset_loader.py:

  • filename-only YAML with two seeds (one unnamed, one with an explicit per-seed name) is loaded, inserted into a real in-memory SQLite memory, then queried by the filename stem and listed by get_seed_dataset_names_async(); the explicit per-seed name is not overwritten;
  • an explicitly named YAML keeps its previous load/insert/query/list behavior.

Evidence:

# parent revision: 1 failed, 8 passed - assert None == 'filename_only'
# this branch:      9 passed

Also ran 289 related unit tests (seed datasets, local loader, dataset e2e contract, seed dataset provider) and 6 local integration tests; ruff check, ruff format --check, ty check, and build_scripts/check_async_suffix.py pass on the changed files.

Not run: the full unit suite, Windows/Python 3.11, and the full pre-commit set. No documentation change: this fixes an internal naming inconsistency, and the documented load/named-query contract is already what the fix makes true.

…ind them

A local YAML without dataset_name got a filename fallback on the provider/SeedDataset only; seed.dataset_name stayed None, so the memory insert persisted NULL and named queries/listings could not find the dataset. Assign the resolved name to unnamed seeds before returning, matching the per-seed assignment used by the generated and jailbreak loaders.

Fixes microsoft#3740
@hannahwestra25 hannahwestra25 self-assigned this Nov 2, 2026
Address review feedback: drop the stale issue reference in the comment/docstring, make both fixture seeds unnamed so DatasetConfiguration.prepare_async exercises the fallback path, and add a prepare_async regression. tests/unit/datasets: 5283 passed; ruff clean.
Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the change that makes unnamed local YAML seeds discoverable through named memory queries.

No open issues 🎉

What changed in this PR

Fixes #3740 by propagating local YAML filename fallback names to unnamed seeds before persistence, making those seeds discoverable through named memory queries.

Changes:

  • Assigns the provider’s resolved name to seeds without a dataset name, preserving existing names.
  • Adds SQLite regression tests for named retrieval, dataset-name listing, scenario preparation, and explicit dataset names.
File Description
tests/​unit/​datasets/​test_local_dataset_loader.py Adds regression coverage for fallback names and explicit-name behavior.
pyrit/​datasets/​seed_datasets/​local/​local_dataset_loader.py Assigns resolved dataset names to unnamed seeds before returning them.

Review effort: Balanced

Add a keyword-only default_dataset_name to SeedDataset.from_yaml_file and
load_seed_dataset_from_yaml. It is used only when the YAML declares neither
dataset_name nor name, so _build_seeds applies its existing fill-only merge
and seeds with their own dataset_name keep it. _LocalDatasetLoader passes its
resolved name instead of re-stamping seeds after loading.

Strengthen the explicit-name loader test with a separately named seed so an
unconditional overwrite of per-seed names fails, and add model-level tests
for the default (fills only unnamed seeds, declared dataset_name/name win,
no-op when omitted).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for contributing :)

@hannahwestra25
hannahwestra25 added this pull request to the merge queue Nov 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 6, 2026
Resolve local dataset loader test imports using public package exports.
@feiiiiii5

Copy link
Copy Markdown
Contributor Author

The only failing check is main-job (macos-latest, 3.11, pyrit, dev_all), in tests/unit/backend/test_chat_scoring.py::test_late_score_does_not_overwrite_changed_settings_async[True-False-scorer] (TimeoutError). This PR only touches dataset/seed loading files; that scoring test is unchanged. The same base job passes on current main (https://github.com/microsoft/PyRIT/actions/runs/42591546534/job/129074094804), and on this head the parametrized scoring test passes locally (8 passed; the two changed test files also pass, 63 passed). This looks like a runner timeout rather than a code failure, but I have not rerun the full 18-minute macOS job.

@hannahwestra25
hannahwestra25 added this pull request to the merge queue Nov 10, 2026
Merged via the queue into microsoft:main with commit 682259d Nov 10, 2026
59 checks passed
Julio César Suástegui (juliosuas) added a commit to juliosuas/PyRIT that referenced this pull request Nov 10, 2026
Brings in microsoft#3663 (reload a sending conversation after another one has
loaded) and microsoft#3772. The ChatWindow conflict is resolved by keeping the
extracted send workflow; microsoft#3663's markConversationLoaded() on the created
conversation moves to the first-send branch of useChatSend, where the
created attack is now routed. Adds a useChatSend test for it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local YAML filename fallback is not assigned to seeds, so the loaded dataset disappears from named queries

4 participants