Repository navigation
FEAT: add FloatScaleFallbackScorer to route abstentions to a second scorer - #2723
Merged
Richard Lundeen (richlundeen) merged 3 commits intoSep 25, 2026
Conversation
…corer Some float-scale scorers decline to call a response and return ScoreStatus.UNDETERMINED, for instance a classifier whose calibrated probability falls inside an abstain band. An abstention is only useful if a caller does something with it, and nothing in the scorer layer did. FloatScaleFallbackScorer is that caller. It scores with a primary scorer first and, only when any returned score is undetermined, scores again with a fallback scorer and returns that result instead. The typical pairing is a cheap primary that covers the bulk of responses and an LLM judge as the fallback, so the judge is paid for only on the uncertain tail. Every returned score records which scorer produced it in score_metadata["resolved_by"]. When the fallback produced it, the primary's rationale is kept under score_metadata["primary_rationale"] and the primary's own metadata is merged in, so nothing the primary said is lost. If the fallback abstains too, the score stays undetermined and its rationale says both declined. A non-applicable primary is propagated silently, a non-applicable fallback leaves the primary's abstention standing, and a fallback error is not swallowed. Follows the shape of the existing wrappers: nested scoring through _score_nested_async, both children in the identifier, conditions and expectation validation delegated to both, observation ids merged. Written with AI assistance.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
approved these changes
Sep 25, 2026
Add true/false fallback support, migrate condition routing, enforce single comparable results, and preserve child evidence and metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
approved these changes
Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Some float-scale scorers decline to call a response and return
ScoreStatus.UNDETERMINED, for instance a classifier whose calibrated probability falls inside an abstain band (#2626). An abstention is only useful if a caller does something with it, and nothing in the scorer layer did: an undetermined score flowed straight through to whatever consumed it.FloatScaleFallbackScoreris that caller. It scores with a primary scorer first and, only when any returned score is undetermined, scores again with a fallback scorer and returns that result instead. The typical pairing is a cheap, fast primary that covers the bulk of responses and an LLM judge as the fallback, so the judge's cost is paid only on the uncertain tail. It is generic: anyFloatScaleScoreras the primary, anyFloatScaleScoreras the fallback.Every returned score records which scorer produced it in
score_metadata["resolved_by"]("primary"or"fallback"). When the fallback produced it, the primary's rationale is kept underscore_metadata["primary_rationale"]and the primary's own metadata (an abstain band, a calibrated probability) is merged in, with the fallback's keys winning on a clash, so nothing the primary said is lost. If the fallback abstains as well, the returned score stays undetermined and its rationale says both scorers declined.Edge behaviour, each pinned by a test: a non-applicable primary (returns
[]) is propagated silently and the fallback is never consulted; a non-applicable fallback leaves the primary's abstention standing; a fallback that raises is not swallowed.It follows the shape of the existing wrappers (
FloatScaleThresholdScorer,TrueFalseInverterScorer): nested scoring through_score_nested_async, both children in the identifier undersub_scorers,matched_conditions,required_conditionsand_validate_expectationdelegated to both, observation ids merged.get_chat_targetreturns the primary's target, or the fallback's when the primary has none, since the fallback is usually the one with an LLM behind it.Tests and Documentation
tests/unit/score/test_float_scale_fallback_scorer.py, ten tests: a complete primary verdict returns without touching the fallback; an abstention routes to the fallback and carries the primary's rationale and metadata; double abstention stays undetermined and says so; fallback metadata keys win on a clash; non-applicable primary; non-applicable fallback; a fallback exception propagates; the identifier names both sub-scorers; constructor rejects a true/false scorer on either side and the same object as both;get_chat_targetfalls through.Verified against upstream main
0ac4526: 10 passed in the new file, 51 across it plus the threshold and inverter wrapper tests.ruff checkandruff formatclean on every touched file.doc/code/scoring/3_combining_scorers.pygains a section on routing abstentions, the wrapper is added to the composition diagram and the wrappers paragraph, and the paired notebook is updated to match.Written with AI assistance.