Repository navigation
TEST: Stop asserting write-only scores by timestamp order - #4371
Merged
Richard Lundeen (richlundeen) merged 1 commit intoNov 18, 2026
Merged
Richard Lundeen (richlundeen) merged 1 commit into
Richard Lundeen (richlundeen) merged 1 commit into
Conversation
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for this test-only assertion update.
No open issues 🎉
What changed in this PR
This test-only change removes a timing-sensitive assertion from backend chat scoring tests, avoiding false failures when concurrent write-only evaluations finish out of delivery order.
Changes:
- Checks that the published automated score belongs to the two stored scores, rather than requiring the latest timestamp.
- Clarifies that dedicated ordering tests verify which delivery’s verdict wins; production code is unchanged.
| File | Description |
|---|---|
tests/unit/backend/test_chat_scoring.py |
Replaces the timestamp-based assertion with score membership and updates the explanatory comment. |
Review effort: Balanced
Justin Song (jsong468)
approved these changes
Nov 17, 2026
Richard Lundeen (richlundeen)
deleted the
test/write-only-score-order-flake
branch
November 18, 2026 06:04
Julio César Suástegui (juliosuas)
added a commit
to juliosuas/PyRIT
that referenced
this pull request
Nov 18, 2026
Resolve tests/unit/scenario/foundry/test_assert_policy.py by keeping main's ASSERT metrics intent from microsoft#4131 plus the PR's exact selection imports and tests. microsoft#4371 landed cleanly in tests/unit/backend/test_chat_scoring.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
test_write_only_send_scores_inherited_side_effects_without_a_response_async(count=2 branch) failed once on the advisory macOS 3.11 CI job even though the correct score was published. This removes the order-sensitive assertion that caused it.test_older_write_only_evaluation_cannot_replace_newer_conversation_score_asyncandtest_equal_time_multipart_write_only_delivery_replaces_earlier_score_async, so no ordering coverage is lost.tests/unit/backend/test_chat_scoring.py, no product code touched.This carries a change that was reviewed on #3757 but missed that PR's merge (commit d15584e on the
fix/write-only-target-false-successbranch of shashank03-dev/PyRIT).