Skip to content

TEST: Stop asserting write-only scores by timestamp order - #4371

Merged
Richard Lundeen (richlundeen) merged 1 commit into
microsoft:mainfrom
richlundeen:test/write-only-score-order-flake
Nov 18, 2026
Merged

Richard Lundeen (richlundeen) merged 1 commit into
microsoft:mainfrom
richlundeen:test/write-only-score-order-flake

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

test_write_only_send_scores_inherited_side_effects_without_a_response_async (count=2 branch) failed once on the advisory macOS 3.11 CI job even though the correct score was published. This removes the order-sensitive assertion that caused it.

  • The assertion required the published automated score to be the one with the latest timestamp. The two deliveries are evaluated concurrently, so the newer delivery's scorer can finish first and its score gets the earlier timestamp. The published score was right, but the timestamp comparison failed.
  • The assertion now only checks that the published score is one of the two stored scores. Which delivery's verdict wins is already pinned by test_older_write_only_evaluation_cannot_replace_newer_conversation_score_async and test_equal_time_multipart_write_only_delivery_replaces_earlier_score_async, so no ordering coverage is lost.
  • Test-only change: three lines in tests/unit/backend/test_chat_scoring.py, no product code touched.

This carries a change that was reviewed on #3757 but missed that PR's merge (commit d15584e on the fix/write-only-target-false-success branch of shashank03-dev/PyRIT).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 21:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for this test-only assertion update.

No open issues 🎉

What changed in this PR

This test-only change removes a timing-sensitive assertion from backend chat scoring tests, avoiding false failures when concurrent write-only evaluations finish out of delivery order.

Changes:

  • Checks that the published automated score belongs to the two stored scores, rather than requiring the latest timestamp.
  • Clarifies that dedicated ordering tests verify which delivery’s verdict wins; production code is unchanged.
File Description
tests/​unit/​backend/​test_chat_scoring.py Replaces the timestamp-based assertion with score membership and updates the explanatory comment.

Review effort: Balanced

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 18, 2026
Merged via the queue into microsoft:main with commit fdeb9c5 Nov 18, 2026
59 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the test/write-only-score-order-flake branch November 18, 2026 06:04
Julio César Suástegui (juliosuas) added a commit to juliosuas/PyRIT that referenced this pull request Nov 18, 2026
Resolve tests/unit/scenario/foundry/test_assert_policy.py by keeping
main's ASSERT metrics intent from microsoft#4131 plus the PR's exact selection
imports and tests. microsoft#4371 landed cleanly in tests/unit/backend/test_chat_scoring.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants