Repository navigation
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 506ade2500
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cast(Tracer, tracer).id_generator = _SavedIdGenerator( | ||
| int(saved["trace_id"], 16), int(saved["span_id"], 16) | ||
| ) |
There was a problem hiding this comment.
Restore the tracer's shared ID generator
In a long-lived worker, UiPathTraceManager reuses the global tracer provider across jobs, so assigning a fixed generator to this cached tracer persists after the resumed evaluation completes. A later non-resume job does not replace the generator because saved is false, causing its Eval Set Run and Evaluation spans to reuse the prior evaluation's trace/span IDs; this can overwrite prior trace data and even give a child the same span ID as its parent. Avoid mutating the shared tracer or restore its original generator after creating the resumed span.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The premise doesn't hold on the SDK we pin. TracerProvider.get_tracer in opentelemetry-sdk 1.39.1 builds a new Tracer on every call, so the generator I set lived and died with that one tracer and a later job never saw it. I still changed it so it doesn't depend on that detail. _reopen now swaps id_generator only around the synchronous start_span call and puts the original back in a finally, so even a provider that caches tracers keeps random ids for the next job.
Tests: test_reopening_leaves_a_cached_tracers_id_generator_alone uses a provider that returns the same tracer every time, and test_fresh_run_after_a_resume_gets_new_span_ids runs a fresh job after a resume and checks that no span id is reused.
🤖 Addressed by Claude Code
There was a problem hiding this comment.
🟡 Changes recommended
Resuming beneath a different active caller span can change the restored trace ID and parent relationship.
1 open issue
What changed in this PR
Restores recording evaluation spans after suspension so resumed scores reach the trace backend.
Changes:
- Adds shared span persistence and restoration, including saved start times.
- Adds resume tracing tests and updates the test tracer.
- Bumps
uipathto2.14.42.
| File | Description |
|---|---|
packages/uipath/uv.lock |
Updates locked package version. |
packages/uipath/tests/cli/eval/test_eval_tracing_integration.py |
Allows additional tracer arguments. |
packages/uipath/tests/cli/eval/test_eval_resume_spans.py |
Tests resumed scores, span identity, and execution IDs. |
packages/uipath/src/uipath/eval/runtime/runtime.py |
Uses the shared resumable-span helper. |
packages/uipath/src/uipath/eval/runtime/_resumable_spans.py |
Persists and reopens recording spans. |
packages/uipath/pyproject.toml |
Bumps the package version. |
Review effort: Balanced
| with tracer.start_as_current_span( | ||
| name, attributes=attributes, start_time=start_time |
There was a problem hiding this comment.
Valid, fixed. The saved record now includes the span's parent. On resume _parent_context restores it explicitly: an empty Context() for a span that was a root, or a remote parent built from the saved ids. Records saved before this change have no parent key and keep the old behavior of using the current context, so a job suspended on the previous version still resumes.
Tests: test_resume_under_a_different_caller_span_keeps_the_original_trace resumes inside an unrelated caller span and checks that the Eval Set Run and Evaluation spans keep their original trace id and parent. test_record_saved_without_parent_still_reopens_the_saved_span covers the old record shape.
🤖 Addressed by Claude Code
|
On resume the runtime re-entered both spans as NonRecordingSpans, so the scores set on them were dropped and the evaluator span lost execution.id. Re-open them as recording spans under the saved ids and start time so the trace upsert replaces the suspended job's empty output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Save the parent context with the span and restore it explicitly on resume, so a resume under a different caller span stays in the original trace. Swap the tracer's id generator only for the start_span call and put it back, so a tracer reused by a later job keeps random ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
506ade2 to
49c9ff4
Compare
🚨 Heads up:
|
With OTEL_SDK_DISABLED the tracer is a NoOpTracer, so the suspend job saved all-zero ids and the resume job crashed setting id_generator. Skip saving invalid span contexts and ignore zero-id records already written by the previous version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🚨 Heads up:
|




When an eval item suspends on a process tool or API workflow tool call and resumes in a second job, the evaluator runs and scores but the scores never reach the trace. The
Evaluationspan has nooutput.scores, theEvaluation Set Runspan stays at{"scores": {}}, and theEvaluator:span is missingexecution.id. Teodor hit this in cloud: items bed87113 and c3f157bb in the shared trace.Root cause
On resume
_restore_parent_spanrebuilt both spans as aNonRecordingSpanfrom the saved ids and entered them withuse_span(..., end_on_exit=False). Because of that:configure_evaluation_spanandconfigure_eval_set_run_spancallset_attributeon a non-recording span, which is a no-op, so the scores are droppedUiPathExecutionTraceProcessorMixinonly copiesexecution.idfrom a recording parent, so the evaluator span gets noneFix
New
_resumable_spans.pywithstart_resumable_span. On resume it starts a real recording span through a tracer whoseid_generatorreturns the saved trace and span id, with the savedstart_time. The LlmOps upsert is keyed on span id, so the resumed job overwrites the suspended job's row with the scored one. The agent spans already rely on this, which is why the agent run span in the trace has its start before the suspend and its end after the resume. With theEvaluationspan recording again,execution.idreaches the evaluator span the normal way.The saved context now stores
start_timeas well. A context saved by an older version still works, its start time just becomes the resume time.The four storage helpers on
UiPathEvalRuntimemoved into the new module, andruntime.pyuses one_resumable_spancall for both spans.Tests
test_eval_resume_spans.pyruns a suspend job and then a resume job against shared in-memory storage and anInMemorySpanExporter. It checks that both spans report scores, keep their original span id, trace id, parent and start time, and that the evaluator span carries the evaluation'sexecution.id. All four failed before the fix. The fake tracer intest_eval_tracing_integration.pynow acceptsstart_time, matching the SDK signature.Not in this PR
rootspan with a new id. Cosmetic, separate change.UPDATE_EVAL_SET_RUNwith empty scores, buthandle_update_eval_set_runskips it because the eval set run id is user provided, so I left it alone.Related: #1949 (empty
AgentRunHistoryfrom the same trace).🤖 Generated with Claude Code