Skip to content

FEAT: Add standalone ASSERT conversation scoring - #3539

Merged
Richard Lundeen (richlundeen) merged 5 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-scoring
Nov 6, 2026
Merged

Richard Lundeen (richlundeen) merged 5 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-scoring

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Add standalone ASSERT policy-violation and overrefusal scoring over retained conversations, with optional category validation and policy-flag correction.

Preserve native scores and evidence, reject unreadable media before judging, and add executed usage examples and pinned source attribution.

Add direct and category-validated policy scoring with retained evidence. Reject unreadable conversation media before judging.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate pinned main 36ad292 without changing the PR's scoring semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise both category orders, conversation and message triggers, and matching or corrected judge verdicts through persisted public scorer results.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 01:29

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the standalone ASSERT conversation-scoring changes.

No open issues 🎉

What changed in this PR

Adds standalone ASSERT policy-violation and overrefusal scoring for retained conversations, composing PyRIT’s existing true/false scorer and conversation wrapper without requiring an ASSERT generator or scenario.

Changes:

  • Adds direct and category-validated judgment modes, with optional policy-verdict correction and retained evidence.
  • Rejects unsupported conversation media before judging and adds tests for validation, persistence, retries, and replay.
  • Adds usage examples and pinned ASSERT source attribution.
File Description
third_party/​assert-provenance.json Records scorer and prompt-template sources and adaptations.
THIRD_PARTY_NOTICES.txt Extends ASSERT attribution to conversation scoring.
tests/​unit/​score/​test_assert_policy_scorer.py Tests scoring contracts, validation, evidence retention, and replay.
pyrit/​score/​true_false/​assert_policy_scorer.py Implements the builder, dimensions, judgment modes, and policy correction.
pyrit/​score/​conversation_scorer.py Adds strict evidence validation and corresponding identifier fields.
pyrit/​score/​__init__.py Exposes ASSERT scoring APIs through lazy exports.
pyrit/​datasets/​score/​true_false_question/​assert_policy_system_prompt.yaml Defines direct failure-evaluation instructions and response schema.
pyrit/​datasets/​score/​true_false_question/​assert_category_policy_system_prompt.yaml Defines category-level evaluation instructions and response schema.
doc/​code/​scoring/​1_true_false_scorers.py Documents scoring behavior and provides standalone usage examples.
doc/​code/​scoring/​1_true_false_scorers.ipynb Adds matching notebook examples with recorded scoring output.

Review effort: Balanced

@romanlutz Roman Lutz (romanlutz) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Can't wait to have it in :-)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the 20 ms polling deadline while allowing database setup to complete on loaded CI runners.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 6, 2026
Merged via the queue into microsoft:main with commit f52a881 Nov 6, 2026
57 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-assert-scoring branch November 6, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants