Skip to content

TEST: Preserve LeakReplay end-to-end coverage - #4241

Merged
Richard Lundeen (richlundeen) merged 3 commits into
microsoft:mainfrom
richlundeen:test-e2e-leak-replay-population-cap
Nov 12, 2026
Merged

Richard Lundeen (richlundeen) merged 3 commits into
microsoft:mainfrom
richlundeen:test-e2e-leak-replay-population-cap

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

LeakReplay's two default techniques and four corpora require eight populations,
but the end-to-end harness currently supplies a limit of one.

  • Set the LeakReplay-specific limit to eight, preserving all default populations.
  • Add a regression that checks the harness arguments against the real scenario
    configuration.

The limit policy and scenario were introduced in #3641 and #3689, respectively.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 09:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test imports the runtime-only pyrit.scenario.garak alias without a targeted suppression, causing an unresolved-import failure in CI’s make ty check.

1 open issue
What changed in this PR

This PR extends the end-to-end harness’s coverage-preserving limit policy to LeakReplay, retaining all eight default technique-and-corpus populations.

Changes:

  • Set LeakReplay’s end-to-end dataset limit to eight.
  • Add a regression checking harness arguments against the real scenario configuration and expected populations.
File Description
tests/​unit/​scenario/​test_scenario_e2e_contract.py Verify LeakReplay harness settings preserve all default populations.
tests/​end_to_end/​test_scenarios.py Add the LeakReplay-specific limit of eight.

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

review:5872374786

Moved the population regression into test_leak_replay.py and added ("garak.leak_replay", 8) to the existing CLI-limit parametrization in a129ac1. The contract module no longer imports the runtime-only alias; scoped tests (56 passed) and ty pass.

Verified on published commit a129ac1dbd619a7490148326a94501be39538ff9.

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 12, 2026
Merged via the queue into microsoft:main with commit ac44bf9 Nov 12, 2026
58 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the test-e2e-leak-replay-population-cap branch November 12, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants