Repository navigation
FEAT: exhaustive tool matching, so ToolsCalled can say "and nothing else" - #4151
Conversation
…lse" ToolsCalled is a subset match, so a run that calls the required tools plus others satisfies it. Add exhaustive=True for an exact match: the named tools ran and no other tool did. With no tools it asks that no tool ran at all. One observed execution of an unnamed tool disproves the match, even from partial evidence; confirming it needs complete, fully correlated evidence. OtelToolCallScorer now downgrades only verdicts that rest on absence when a request carries no trace link, which leaves subset behaviour unchanged. MessageToolCallScorer can refute an exhaustive match from a stored execution but never confirm one. The flag is left out of serialization while false, so stored subset conditions keep their form and expectation fingerprint (pinned in a test).
There was a problem hiding this comment.
🟡 Changes recommended
The new async test helper lacks the required _async suffix and Score return annotation, violating mandatory repository conventions.
1 open issue
What changed in this PR
Adds exhaustive tool matching to PyRIT’s scoring conditions, allowing checks that required tools—and no others—ran, including checks that no tools ran.
Changes:
- Adds
ToolsCalled.exhaustivewhile preserving existing subset serialization and fingerprints. - Updates trace and message scorers to distinguish observed extra calls from absence-based conclusions.
- Adds validation, scoring, replay, and correlation tests, plus documentation.
| File | Description |
|---|---|
tests/unit/score/test_otel_tool_call_scorer.py |
Tests exhaustive matching, evidence completeness, and replay. |
tests/unit/score/test_message_trace_scoring.py |
Tests exhaustive verdicts with unlinked requests. |
tests/unit/score/test_message_tool_call_scorer.py |
Tests exhaustive scoring from stored executions. |
tests/unit/models/test_condition.py |
Tests validation, round trips, and fingerprint compatibility. |
pyrit/score/true_false/otel_tool_call_scorer.py |
Adds exhaustive matching and absence-aware correlation handling. |
pyrit/score/true_false/message_tool_call_scorer.py |
Refutes exhaustive matches when extra executions are recorded. |
pyrit/models/score/condition.py |
Defines the exhaustive flag and permits empty exhaustive requirements. |
doc/code/scoring/5_tool_call_scorer.py |
Explains exhaustive matching and evidence requirements. |
doc/code/scoring/5_tool_call_scorer.ipynb |
Adds the corresponding notebook documentation. |
Review effort: Balanced
Follow the repository convention for async helpers: add the _async suffix and the Score return annotation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The helper is named _score_exhaustive_async, returns Score, and all three calls use the suffixed name. Verified in published commit 95f060c. Verified on published commit |
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Drops the unreachable subset branch from _rationale, which is only called for exhaustive conditions, and builds the subset rationale only on the subset path.
|
Thanks for the review and the error-acquisition fix. I pushed d87dc0e for the diff-cover gate: |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reconciles the sibling-scorer refactor with main's tool argument matching (microsoft#4150), exhaustive matching (microsoft#4151), and the framework.md rewrite (microsoft#4175): - The unlinked-request downgrade from microsoft#4151 asked _match_tools_called whether the verdict rests on absence, which hardcoded ToolsCalled in what is now the shared base class and would reject every OtelToolCallOrderScorer scoring call. Each leaf now answers through an abstract _verdict_rests_on_absence hook: the presence scorer keeps main's rule, and the order scorer downgrades exactly its False verdicts, which rest on complete coverage, as before the merge. - framework.md takes main's concise rewrite, which no longer names individual scorers. - The tool-call scorer doc merges both new sections and the notebook was regenerated from the paired script and executed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Description
Answers open question 4 in the scorer contract design: can an expectation express "and nothing else"?
ToolsCalledis a subset match, so a run that calls the required tools plus others still satisfies it. This addsexhaustive=Truefor an exact match: the named tools ran and no other tool did. An exhaustive condition with no tools asks that no tool ran at all, which is the shape a least-privilege or confirmation-gate check needs.The evidence rules follow the existing asymmetry. One observed execution of an unnamed tool disproves the match, even from partial evidence. Confirming it needs complete, fully correlated evidence.
OtelToolCallScorernow downgrades only verdicts that rest on absence when a request carries no trace link, so subset behaviour is unchanged.MessageToolCallScorercan refute an exhaustive match from a stored execution of an unnamed tool but never confirm one, since stored messages cannot rule out other tools.The flag is left out of serialization while false, so stored subset conditions keep their form and expectation fingerprint, pinned in a test against main. The tool-call scorer docs gain a short section on exhaustive matching.
How it was tested
pytest tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack -n 2: 11573 passed, 65 skipped.MessageToolCallScorerturns 1 test red; restoring the old "at least one tool" validator turns 7 red.