Skip to content

feat(guardrails): report provider errors on guardrail validation results [AL-673] - #1956

Merged
apetraru-uipath merged 1 commit into
mainfrom
fix/al-673-guardrail-error-type
Nov 2, 2026
Merged

apetraru-uipath merged 1 commit into
mainfrom
fix/al-673-guardrail-error-type

Conversation

@apetraru-uipath

@apetraru-uipath apetraru-uipath commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

The guardrail validate API answers PROVIDER_ERROR, INTERNAL_ERROR or RATE_LIMIT_EXCEEDED when it could not evaluate the input. evaluate_guardrail turned these into VALIDATION_FAILED through its unknown-result fallback, so callers could not tell a real violation from a validation that never ran.

Prod impact (SRE-687087): on 2026-10-22, an LLM-as-Judge model rejected temperature with a 400. 90 eval runs were reported as guardrail violations.

Changes:

  • GuardrailValidationResult.error_type (alias errorType) uses a new GuardrailValidationErrorType enum: provider_error, internal_error or rate_limit_exceeded.
    • It is set only for those backend results, matched case-insensitively.
    • It is None for every other result.
  • result stays validation_failed, so the guardrail still fails closed. No consumer changes are needed:
    • Every existing == / != VALIDATION_FAILED check, here and in uipath-langchain (middlewares, adapter, escalate action, guardrail nodes), still fires its action.
    • Adding new values to GuardrailValidationResultType was rejected on purpose. Those same checks would have started letting runs through, and released uipath-langchain versions would pick up a newer platform.
  • Logging: an error result and an unknown result each log a warning.
  • Imports: the enum is exported from uipath.core.guardrails only. It is not added to the uipath.platform.guardrails re-export block, because that block is marked for removal in 2.3.0.

Versions:

  • uipath-core 0.5.34.
  • uipath-platform 0.2.42, which requires core ≥ 0.5.34.
  • The uv.lock files are refreshed.

Follow-ups:

  • uipath-langchain will label these failures in the guardrail validation details ("Guardrail could not be evaluated (provider_error): …") once this is released.
  • The backend fix for the root cause is UiPath/Agents#6797.

Ticket: AL-673

How has this been tested?

  • uipath-core guardrail tests: 51 passed. The full suite is 246 passed. They cover:
    • error_type defaults to None.
    • The errorType alias round-trips.
    • The enum values.
  • uipath-platform service tests: 59 passed. The full suite is 1898 passed and 7 skipped (the live LLM tests). They cover:
    • Each error result gives validation_failed, the matching error_type, and the backend details kept as reason. This holds for lowercase results too.
    • An unknown result gives validation_failed, error_type=None and a warning.
    • PASSED and VALIDATION_FAILED give error_type=None.
  • ruff check, ruff format --check and mypy are clean on both packages.
  • check_version_uniqueness.py and check_dependency_version_bumps.py pass, and uv lock --locked is current.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 25, 2026 03:59
@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-runtime test:uipath-integrations labels Oct 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The additive metadata preserves existing failure handling, has focused regression coverage, and leaves no unresolved blocking findings.

No open issues 🎉

What changed in this PR

This PR adds error metadata to guardrail validation results so callers can distinguish service failures from actual violations while preserving existing fail-closed behavior.

Changes:

  • Adds and exports GuardrailValidationErrorType with an optional error_type result field.
  • Maps backend errors case-insensitively and logs warnings for errors and unknown results.
  • Adds regression tests and updates package versions, dependency requirements, and lockfiles.
File Description
packages/​uipath/​uv.lock Updates locked core and platform versions.
packages/​uipath-platform/​uv.lock Updates locked core and platform versions.
packages/​uipath-platform/​tests/​services/​test_guardrails_service.py Tests error mapping, warnings, and preserved failure behavior.
packages/​uipath-platform/​src/​uipath/​platform/​guardrails/​_guardrails_service.py Classifies backend errors and attaches metadata while failing closed.
packages/​uipath-platform/​pyproject.toml Bumps platform version and minimum core dependency.
packages/​uipath-core/​uv.lock Updates the locked core version.
packages/​uipath-core/​tests/​guardrails/​test_guardrails_models.py Tests enum values, defaults, aliases, and serialization.
packages/​uipath-core/​src/​uipath/​core/​guardrails/​guardrails.py Defines the error enum and optional result field.
packages/​uipath-core/​src/​uipath/​core/​guardrails/​__init__.py Exports the new error enum.
packages/​uipath-core/​pyproject.toml Bumps core to version 0.5.34.

Review effort: Balanced

…lts [AL-673]

The guardrail validate API answers PROVIDER_ERROR, INTERNAL_ERROR or
RATE_LIMIT_EXCEEDED when it could not evaluate the input, for example when
an LLM-as-Judge model rejects the request. evaluate_guardrail turned these
into VALIDATION_FAILED through its unknown-result fallback, so callers could
not tell a real violation from a failed evaluation (SRE-687087: judge models
that deprecate `temperature` answered 400 and 90 eval runs were blocked).

GuardrailValidationResult gains an additive error_type
(GuardrailValidationErrorType) that is set for those results. result stays
validation_failed, so every existing VALIDATION_FAILED check keeps firing
its action (fail closed); error_type only says why the validation did not
complete. Error and unknown results are now logged as warnings.

Bumps uipath-core to 0.5.34 and uipath-platform to 0.2.42 (requiring
uipath-core>=0.5.34) and refreshes the uv.lock files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apetraru-uipath
apetraru-uipath force-pushed the fix/al-673-guardrail-error-type branch from 0086f38 to bcc7c95 Compare November 2, 2026 11:00
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

🚨 Heads up: uipath-integrations cross-tests are FAILING 🚨

Your changes may break one or more integrations in uipath-integrations-python:

  • uipath-openai-agents
  • uipath-google-adk
  • uipath-agent-framework
  • uipath-llamaindex
  • uipath-pydantic-ai

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

@apetraru-uipath
apetraru-uipath merged commit 2eb6827 into main Nov 2, 2026
149 of 186 checks passed
@apetraru-uipath
apetraru-uipath deleted the fix/al-673-guardrail-error-type branch November 2, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-integrations test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants