Skip to content

fix(eval): report scores on resumed Evaluation and Eval Set Run spans - #1950

Open
Chibionos wants to merge 3 commits into
mainfrom
fix/eval-resume-scores
Open

Chibionos wants to merge 3 commits into
mainfrom
fix/eval-resume-scores

Conversation

@Chibionos

Copy link
Copy Markdown
Contributor

When an eval item suspends on a process tool or API workflow tool call and resumes in a second job, the evaluator runs and scores but the scores never reach the trace. The Evaluation span has no output.scores, the Evaluation Set Run span stays at {"scores": {}}, and the Evaluator: span is missing execution.id. Teodor hit this in cloud: items bed87113 and c3f157bb in the shared trace.

Root cause

On resume _restore_parent_span rebuilt both spans as a NonRecordingSpan from the saved ids and entered them with use_span(..., end_on_exit=False). Because of that:

  • configure_evaluation_span and configure_eval_set_run_span call set_attribute on a non-recording span, which is a no-op, so the scores are dropped
  • the span never ends, so nothing is upserted under the original id and the backend keeps the empty row the suspended job closed
  • UiPathExecutionTraceProcessorMixin only copies execution.id from a recording parent, so the evaluator span gets none

Fix

New _resumable_spans.py with start_resumable_span. On resume it starts a real recording span through a tracer whose id_generator returns the saved trace and span id, with the saved start_time. The LlmOps upsert is keyed on span id, so the resumed job overwrites the suspended job's row with the scored one. The agent spans already rely on this, which is why the agent run span in the trace has its start before the suspend and its end after the resume. With the Evaluation span recording again, execution.id reaches the evaluator span the normal way.

The saved context now stores start_time as well. A context saved by an older version still works, its start time just becomes the resume time.

The four storage helpers on UiPathEvalRuntime moved into the new module, and runtime.py uses one _resumable_span call for both spans.

Tests

test_eval_resume_spans.py runs a suspend job and then a resume job against shared in-memory storage and an InMemorySpanExporter. It checks that both spans report scores, keep their original span id, trace id, parent and start time, and that the evaluator span carries the evaluation's execution.id. All four failed before the fix. The fake tracer in test_eval_tracing_integration.py now accepts start_time, matching the SDK signature.

Not in this PR

  • 8b4c11a5 suspended and never resumed, its agent run and tool call spans have no end time. Nothing in the SDK drives the resume, that is on the orchestrator side.
  • Each resume still opens a second agent root span with a new id. Cosmetic, separate change.
  • In cloud the suspend job still publishes UPDATE_EVAL_SET_RUN with empty scores, but handle_update_eval_set_run skips it because the eval set run id is user provided, so I left it alone.

Related: #1949 (empty AgentRunHistory from the same trace).

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 19, 2026 17:51
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T17:54:36.817013Z 506ade2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 506ade2500

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +44 to +46
cast(Tracer, tracer).id_generator = _SavedIdGenerator(
int(saved["trace_id"], 16), int(saved["span_id"], 16)
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore the tracer's shared ID generator

In a long-lived worker, UiPathTraceManager reuses the global tracer provider across jobs, so assigning a fixed generator to this cached tracer persists after the resumed evaluation completes. A later non-resume job does not replace the generator because saved is false, causing its Eval Set Run and Evaluation spans to reuse the prior evaluation's trace/span IDs; this can overwrite prior trace data and even give a child the same span ID as its parent. Avoid mutating the shared tracer or restore its original generator after creating the resumed span.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The premise doesn't hold on the SDK we pin. TracerProvider.get_tracer in opentelemetry-sdk 1.39.1 builds a new Tracer on every call, so the generator I set lived and died with that one tracer and a later job never saw it. I still changed it so it doesn't depend on that detail. _reopen now swaps id_generator only around the synchronous start_span call and puts the original back in a finally, so even a provider that caches tracers keeps random ids for the next job.

Tests: test_reopening_leaves_a_cached_tracers_id_generator_alone uses a provider that returns the same tracer every time, and test_fresh_run_after_a_resume_gets_new_span_ids runs a fresh job after a resume and checks that no span id is reused.

🤖 Addressed by Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Resuming beneath a different active caller span can change the restored trace ID and parent relationship.

1 open issue
What changed in this PR

Restores recording evaluation spans after suspension so resumed scores reach the trace backend.

Changes:

  • Adds shared span persistence and restoration, including saved start times.
  • Adds resume tracing tests and updates the test tracer.
  • Bumps uipath to 2.14.42.
File Description
packages/​uipath/​uv.lock Updates locked package version.
packages/​uipath/​tests/​cli/​eval/​test_eval_tracing_integration.py Allows additional tracer arguments.
packages/​uipath/​tests/​cli/​eval/​test_eval_resume_spans.py Tests resumed scores, span identity, and execution IDs.
packages/​uipath/​src/​uipath/​eval/​runtime/​runtime.py Uses the shared resumable-span helper.
packages/​uipath/​src/​uipath/​eval/​runtime/​_resumable_spans.py Persists and reopens recording spans.
packages/​uipath/​pyproject.toml Bumps the package version.

Review effort: Balanced

Comment on lines +49 to +50
with tracer.start_as_current_span(
name, attributes=attributes, start_time=start_time

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, fixed. The saved record now includes the span's parent. On resume _parent_context restores it explicitly: an empty Context() for a span that was a root, or a remote parent built from the saved ids. Records saved before this change have no parent key and keep the old behavior of using the current context, so a job suspended on the previous version still resumes.

Tests: test_resume_under_a_different_caller_span_keeps_the_original_trace resumes inside an unrelated caller span and checks that the Eval Set Run and Evaluation spans keep their original trace id and parent. test_record_saved_without_parent_still_reopens_the_saved_span covers the old record shape.

🤖 Addressed by Claude Code

@sonarqubecloud

Copy link
Copy Markdown

@Chibionos
Chibionos requested a review from AAgnihotry November 18, 2026 08:40
Chibionos and others added 2 commits November 18, 2026 18:38
On resume the runtime re-entered both spans as NonRecordingSpans, so the
scores set on them were dropped and the evaluator span lost execution.id.
Re-open them as recording spans under the saved ids and start time so the
trace upsert replaces the suspended job's empty output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Save the parent context with the span and restore it explicitly on
resume, so a resume under a different caller span stays in the original
trace. Swap the tracer's id generator only for the start_span call and
put it back, so a tracer reused by a later job keeps random ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Chibionos
Chibionos force-pushed the fix/eval-resume-scores branch from 506ade2 to 49c9ff4 Compare November 19, 2026 02:43
@github-actions

Copy link
Copy Markdown

🚨 Heads up: uipath-integrations cross-tests are FAILING 🚨

Your changes may break one or more integrations in uipath-integrations-python:

  • uipath-openai-agents
  • uipath-google-adk
  • uipath-agent-framework
  • uipath-llamaindex
  • uipath-pydantic-ai

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

With OTEL_SDK_DISABLED the tracer is a NoOpTracer, so the suspend job
saved all-zero ids and the resume job crashed setting id_generator.
Skip saving invalid span contexts and ignore zero-id records already
written by the previous version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-integrations test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants