Skip to content

FEAT: add FloatScaleFallbackScorer to route abstentions to a second scorer - #2723

Merged
Richard Lundeen (richlundeen) merged 3 commits into
microsoft:mainfrom
WatchTree-19:feat/float-scale-fallback-scorer
Sep 25, 2026
Merged

Richard Lundeen (richlundeen) merged 3 commits into
microsoft:mainfrom
WatchTree-19:feat/float-scale-fallback-scorer

Conversation

@WatchTree-19

Copy link
Copy Markdown
Contributor

Description

Some float-scale scorers decline to call a response and return ScoreStatus.UNDETERMINED, for instance a classifier whose calibrated probability falls inside an abstain band (#2626). An abstention is only useful if a caller does something with it, and nothing in the scorer layer did: an undetermined score flowed straight through to whatever consumed it.

FloatScaleFallbackScorer is that caller. It scores with a primary scorer first and, only when any returned score is undetermined, scores again with a fallback scorer and returns that result instead. The typical pairing is a cheap, fast primary that covers the bulk of responses and an LLM judge as the fallback, so the judge's cost is paid only on the uncertain tail. It is generic: any FloatScaleScorer as the primary, any FloatScaleScorer as the fallback.

Every returned score records which scorer produced it in score_metadata["resolved_by"] ("primary" or "fallback"). When the fallback produced it, the primary's rationale is kept under score_metadata["primary_rationale"] and the primary's own metadata (an abstain band, a calibrated probability) is merged in, with the fallback's keys winning on a clash, so nothing the primary said is lost. If the fallback abstains as well, the returned score stays undetermined and its rationale says both scorers declined.

Edge behaviour, each pinned by a test: a non-applicable primary (returns []) is propagated silently and the fallback is never consulted; a non-applicable fallback leaves the primary's abstention standing; a fallback that raises is not swallowed.

It follows the shape of the existing wrappers (FloatScaleThresholdScorer, TrueFalseInverterScorer): nested scoring through _score_nested_async, both children in the identifier under sub_scorers, matched_conditions, required_conditions and _validate_expectation delegated to both, observation ids merged. get_chat_target returns the primary's target, or the fallback's when the primary has none, since the fallback is usually the one with an LLM behind it.

Tests and Documentation

tests/unit/score/test_float_scale_fallback_scorer.py, ten tests: a complete primary verdict returns without touching the fallback; an abstention routes to the fallback and carries the primary's rationale and metadata; double abstention stays undetermined and says so; fallback metadata keys win on a clash; non-applicable primary; non-applicable fallback; a fallback exception propagates; the identifier names both sub-scorers; constructor rejects a true/false scorer on either side and the same object as both; get_chat_target falls through.

Verified against upstream main 0ac4526: 10 passed in the new file, 51 across it plus the threshold and inverter wrapper tests. ruff check and ruff format clean on every touched file.

doc/code/scoring/3_combining_scorers.py gains a section on routing abstentions, the wrapper is added to the composition diagram and the wrappers paragraph, and the paired notebook is updated to match.

Written with AI assistance.

…corer

Some float-scale scorers decline to call a response and return
ScoreStatus.UNDETERMINED, for instance a classifier whose calibrated
probability falls inside an abstain band. An abstention is only useful if
a caller does something with it, and nothing in the scorer layer did.

FloatScaleFallbackScorer is that caller. It scores with a primary scorer
first and, only when any returned score is undetermined, scores again with
a fallback scorer and returns that result instead. The typical pairing is
a cheap primary that covers the bulk of responses and an LLM judge as the
fallback, so the judge is paid for only on the uncertain tail.

Every returned score records which scorer produced it in
score_metadata["resolved_by"]. When the fallback produced it, the primary's
rationale is kept under score_metadata["primary_rationale"] and the
primary's own metadata is merged in, so nothing the primary said is lost.
If the fallback abstains too, the score stays undetermined and its
rationale says both declined. A non-applicable primary is propagated
silently, a non-applicable fallback leaves the primary's abstention
standing, and a fallback error is not swallowed.

Follows the shape of the existing wrappers: nested scoring through
_score_nested_async, both children in the identifier, conditions and
expectation validation delegated to both, observation ids merged.

Written with AI assistance.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add true/false fallback support, migrate condition routing, enforce single comparable results, and preserve child evidence and metadata.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Sep 25, 2026
Merged via the queue into microsoft:main with commit 39366f9 Sep 25, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants