Skip to content

FEAT: exhaustive tool matching, so ToolsCalled can say "and nothing else" - #4151

Merged
Richard Lundeen (richlundeen) merged 11 commits into
microsoft:mainfrom
WatchTree-19:feat/exhaustive-tool-matching
Nov 14, 2026
Merged

Richard Lundeen (richlundeen) merged 11 commits into
microsoft:mainfrom
WatchTree-19:feat/exhaustive-tool-matching

Conversation

@WatchTree-19

Copy link
Copy Markdown
Contributor

Description

Answers open question 4 in the scorer contract design: can an expectation express "and nothing else"?

ToolsCalled is a subset match, so a run that calls the required tools plus others still satisfies it. This adds exhaustive=True for an exact match: the named tools ran and no other tool did. An exhaustive condition with no tools asks that no tool ran at all, which is the shape a least-privilege or confirmation-gate check needs.

The evidence rules follow the existing asymmetry. One observed execution of an unnamed tool disproves the match, even from partial evidence. Confirming it needs complete, fully correlated evidence. OtelToolCallScorer now downgrades only verdicts that rest on absence when a request carries no trace link, so subset behaviour is unchanged. MessageToolCallScorer can refute an exhaustive match from a stored execution of an unnamed tool but never confirm one, since stored messages cannot rule out other tools.

The flag is left out of serialization while false, so stored subset conditions keep their form and expectation fingerprint, pinned in a test against main. The tool-call scorer docs gain a short section on exhaustive matching.

How it was tested

  • Rebased on main 935afa8: pytest tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack -n 2: 11573 passed, 65 skipped.
  • ruff check and ruff format clean; ty shows no new diagnostics on the touched files.
  • Mutations: dropping the unnamed-tool refutation in MessageToolCallScorer turns 1 test red; restoring the old "at least one tool" validator turns 7 red.

…lse"

ToolsCalled is a subset match, so a run that calls the required tools plus
others satisfies it. Add exhaustive=True for an exact match: the named tools
ran and no other tool did. With no tools it asks that no tool ran at all.

One observed execution of an unnamed tool disproves the match, even from
partial evidence; confirming it needs complete, fully correlated evidence.
OtelToolCallScorer now downgrades only verdicts that rest on absence when a
request carries no trace link, which leaves subset behaviour unchanged.
MessageToolCallScorer can refute an exhaustive match from a stored execution
but never confirm one.

The flag is left out of serialization while false, so stored subset
conditions keep their form and expectation fingerprint (pinned in a test).
Copilot AI balanced review requested due to automatic review settings November 10, 2026 15:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new async test helper lacks the required _async suffix and Score return annotation, violating mandatory repository conventions.

1 open issue
What changed in this PR

Adds exhaustive tool matching to PyRIT’s scoring conditions, allowing checks that required tools—and no others—ran, including checks that no tools ran.

Changes:

  • Adds ToolsCalled.exhaustive while preserving existing subset serialization and fingerprints.
  • Updates trace and message scorers to distinguish observed extra calls from absence-based conclusions.
  • Adds validation, scoring, replay, and correlation tests, plus documentation.
File Description
tests/​unit/​score/​test_otel_tool_call_scorer.py Tests exhaustive matching, evidence completeness, and replay.
tests/​unit/​score/​test_message_trace_scoring.py Tests exhaustive verdicts with unlinked requests.
tests/​unit/​score/​test_message_tool_call_scorer.py Tests exhaustive scoring from stored executions.
tests/​unit/​models/​test_condition.py Tests validation, round trips, and fingerprint compatibility.
pyrit/​score/​true_false/​otel_tool_call_scorer.py Adds exhaustive matching and absence-aware correlation handling.
pyrit/​score/​true_false/​message_tool_call_scorer.py Refutes exhaustive matches when extra executions are recorded.
pyrit/​models/​score/​condition.py Defines the exhaustive flag and permits empty exhaustive requirements.
doc/​code/​scoring/​5_tool_call_scorer.py Explains exhaustive matching and evidence requirements.
doc/​code/​scoring/​5_tool_call_scorer.ipynb Adds the corresponding notebook documentation.

Review effort: Balanced

Follow the repository convention for async helpers: add the _async
suffix and the Score return annotation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor

review:5844162192

The helper is named _score_exhaustive_async, returns Score, and all three calls use the suffixed name. Verified in published commit 95f060c.

Verified on published commit 95f060c4def0ec9e7beaafa7f5443aac85a13ef1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Drops the unreachable subset branch from _rationale, which is only called
for exhaustive conditions, and builds the subset rationale only on the
subset path.
@WatchTree-19

Copy link
Copy Markdown
Contributor Author

Thanks for the review and the error-acquisition fix. I pushed d87dc0e for the diff-cover gate: _rationale is only called for exhaustive conditions, so its subset branch could never run and is gone, the subset rationale is now built only on the subset path, and the replay test gains the case where a required tool is missing. No behaviour change. The macOS 3.13 failure was the Copilot target timeout, unrelated to this branch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Nov 14, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 14, 2026
Merged via the queue into microsoft:main with commit eb8045a Nov 14, 2026
63 checks passed
Roman Lutz (romanlutz) added a commit to WatchTree-19/PyRIT that referenced this pull request Nov 15, 2026
Reconciles the sibling-scorer refactor with main's tool argument
matching (microsoft#4150), exhaustive matching (microsoft#4151), and the framework.md
rewrite (microsoft#4175):

- The unlinked-request downgrade from microsoft#4151 asked _match_tools_called
  whether the verdict rests on absence, which hardcoded ToolsCalled in
  what is now the shared base class and would reject every
  OtelToolCallOrderScorer scoring call. Each leaf now answers through
  an abstract _verdict_rests_on_absence hook: the presence scorer keeps
  main's rule, and the order scorer downgrades exactly its False
  verdicts, which rest on complete coverage, as before the merge.
- framework.md takes main's concise rewrite, which no longer names
  individual scorers.
- The tool-call scorer doc merges both new sections and the notebook
  was regenerated from the paired script and executed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants