Skip to content

fix: resolve OpenTelemetry tracing architecture issues - #1197

Merged
AAgnihotry merged 7 commits into
mainfrom
fix/tracing-architecture-issues
Jan 26, 2026
Merged

AAgnihotry merged 7 commits into
mainfrom
fix/tracing-architecture-issues

Conversation

@Chibionos

@Chibionos Chibionos commented Jan 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The current tracing implementation has several architectural issues that cause duplicate span exports and violate OpenTelemetry design principles:

  1. Duplicate Span Exports: Same LlmOpsHttpExporter instance added to trace pipeline multiple times in cli_eval.py, causing spans to be exported 2x on start and 4x on end
  2. Filtering in Wrong Layer: LlmOpsHttpExporter._should_drop_span() implements filtering logic (WHAT to send) in the exporter (should only handle HOW to send)
  3. Mutable Exporter State: trace_id set in multiple places after exporter creation, causing potential concurrency issues
  4. Tight Coupling: Hardcoded entrypoint == "agent.json" checks create fragile dependencies

Solution

Separation of Concerns: Move filtering from exporters to processor boundary layer

Key Changes

1. Created FilteringSpanExporter Wrapper

  • New class that wraps any exporter with filtering logic
  • Follows pattern from uipath-agents-python
  • Enables composition: mix any filter with any exporter
# Before: Filtering inside exporter
exporter = LlmOpsHttpExporter(is_low_code=True)  # ❌

# After: Filtering at processor boundary
base_exporter = LlmOpsHttpExporter()
filtered = FilteringSpanExporter(base_exporter, filter_fn)  # ✅

2. Cleaned Up LlmOpsHttpExporter

  • Removed is_low_code parameter
  • Removed _should_drop_span() method
  • Removed filtering logic from export() and upsert_span()
  • Now focuses solely on HTTP transport

3. Fixed Duplicate Exports in cli_eval.py

# Before: Added twice ❌
trace_manager.add_span_exporter(job_exporter)  # First time
processor = LiveTrackingSpanProcessor(job_exporter)
trace_manager.add_span_processor(processor)  # Second time (same exporter!)

# After: Added once ✅
processor = LiveTrackingSpanProcessor(job_exporter)
trace_manager.add_span_processor(processor)

4. Fixed trace_id Mutations

# Before: Mutable state ❌
exporter = LlmOpsHttpExporter()
exporter.trace_id = some_id  # Mutation after creation

# After: Immutable construction ✅
exporter = LlmOpsHttpExporter(trace_id=some_id)

5. Removed agent.json Hacks

# Before: Tight coupling ❌
is_low_code = entrypoint == "agent.json"
exporter = LlmOpsHttpExporter(is_low_code=is_low_code)

# After: Clean abstraction ✅
exporter = LlmOpsHttpExporter()

Benefits

✅ Correct Architecture: Processors filter (WHAT), exporters export (HOW)
✅ No Duplicates: Each span exported exactly once
✅ Immutable State: No post-construction mutations
✅ Loose Coupling: Removed hardcoded filename checks
✅ Extensible: FilteringSpanExporter reusable for any filtering needs

Testing

  • ✅ All ruff linting checks pass
  • ✅ No breaking changes to public API
  • ✅ Backward compatible for non-filtered exports

Files Changed

  • src/uipath/tracing/_otel_exporters.py - Added FilteringSpanExporter, cleaned up LlmOpsHttpExporter
  • src/uipath/_cli/cli_eval.py - Fixed duplicate exports and trace_id mutations
  • src/uipath/_cli/cli_run.py - Removed agent.json hack
  • src/uipath/_cli/_evals/_progress_reporter.py - Removed trace_id mutation
  • src/uipath/tracing/__init__.py - Exported new FilteringSpanExporter class

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.6.9.dev1011974324",

  # Any version from PR
  "uipath>=2.6.9.dev1011970000,<2.6.9.dev1011980000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.6.9.dev1011970000,<2.6.9.dev1011980000",
]

Screenshots

Low code studio web
image

coded orchestrator
image

Coded studioweb

image

@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Jan 26, 2026

@JosephMar JosephMar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saksharthakkar please review

@cristipufu cristipufu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm insisting on these things because we're trying to build apps around the current runtime contracts (uipath dev, flow, etc).
For example, if we filter spans at exporter level, the trace will look completely different with other exporters (json, flow, dev)

Comment thread src/uipath/_cli/cli_debug.py Outdated
trace_manager.add_span_exporter(
LlmOpsHttpExporter(is_low_code=is_low_code)
)
trace_manager.add_span_exporter(LlmOpsHttpExporter())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use this for span filtering: https://github.com/UiPath/uipath-agents-python/pull/166/changes#diff-b71276198b4124a812c5238f4a735e0ed775e2a2fc619a1d6c0f3eacdeb47559R109

factory_settings = await factory.get_settings()
trace_settings = factory_settings.trace_settings if factory_settings else None
trace_manager.add_span_exporter(LlmOpsHttpExporter(), settings=trace_settings)

Comment thread src/uipath/tracing/_otel_exporters.py Resolved
Comment thread src/uipath/_cli/cli_debug.py Resolved

@AAgnihotry AAgnihotry left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please verfy the agent run spans are not being dropped in filtering.

@AAgnihotry
AAgnihotry force-pushed the fix/tracing-architecture-issues branch 2 times, most recently from dce737d to f26f958 Compare January 26, 2026 19:37
@AAgnihotry
AAgnihotry self-requested a review January 26, 2026 19:58
Chibi Vikram and others added 5 commits January 26, 2026 12:00
This commit addresses critical architectural issues in the tracing system
identified by the team, focusing on proper separation of concerns between
span processors and exporters.

- New `FilteringSpanExporter` wrapper class in `_otel_exporters.py`
- Implements filtering at the exporter/processor boundary layer
- Follows the pattern from uipath-agents-python
- Exporters now only handle HOW to send (HTTP, file, format)
- Filtering logic determines WHAT to send (moved out of LlmOpsHttpExporter)

- Removed `is_low_code` parameter from `__init__()`
- Removed `_should_drop_span()` method entirely
- Removed filtering logic from `export()` and `upsert_span()` methods
- Exporter now focuses solely on HTTP transport to LLMOps

- Removed duplicate `trace_manager.add_span_exporter(job_exporter)` call
- Now exporter is added ONLY via LiveTrackingSpanProcessor
- Prevents spans from being exported 2x on start, 4x on end
- Fixed studio_web_tracking_exporter to pass trace_id during construction

- Removed `is_low_code = entrypoint == "agent.json"` from cli_run.py
- Simplified to `LlmOpsHttpExporter()` without parameters
- Reduces tight coupling and fragile filename checks

- cli_eval.py: Pass trace_id during exporter construction, not mutation
- _progress_reporter.py: Removed `self.spans_exporter.trace_id = ...` mutation
- Spans already have trace IDs set from execution context

1. **Proper separation of concerns**: Processors filter (WHAT), exporters export (HOW)
2. **No duplicate exports**: Each span exported exactly once
3. **Immutable exporter state**: No post-construction mutations
4. **Cleaner abstractions**: Removed hardcoded filename checks
5. **Extensible design**: FilteringSpanExporter can be reused for any filtering needs

- All ruff linting checks pass
- No type errors introduced
- Changes maintain backward compatibility for non-filtered exports

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Remove **kwargs from LlmOpsHttpExporter.__init__ to fix TypeError
- Remove is_low_code parameter that was deleted from LlmOpsHttpExporter
- Remove obsolete test classes TestSpanFiltering and TestSpanFilteringByAgentType
- Add new TestFilteringSpanExporter class to test the new wrapper pattern
- Update TestUpsertSpan fixture to remove is_low_code parameter
- Add FilteringSpanExporter to test imports

This fixes the mypy type checking errors and test failures in CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@AAgnihotry
AAgnihotry force-pushed the fix/tracing-architecture-issues branch from bb9e313 to c846016 Compare January 26, 2026 20:01
@AAgnihotry
AAgnihotry dismissed their stale review January 26, 2026 20:08

addressing the concerns

@AAgnihotry AAgnihotry added the build:dev Create a dev build from the pr label Jan 26, 2026
):
self.exporter = exporter
self.span_status = SpanStatus
self.settings = settings

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this used?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Equivalent code:

We are using it on the next line

 if settings and settings.trace_settings:
     self.span_filter = settings.trace_settings.span_filter
 else:
     self.span_filter = None
    ```

@saksharthakkar saksharthakkar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a comment for unused settings... other than that please test E2E to make sure you're getting eval traces as expected.

@AAgnihotry
AAgnihotry merged commit c3f7d5b into main Jan 26, 2026
94 of 100 checks passed
@AAgnihotry
AAgnihotry deleted the fix/tracing-architecture-issues branch January 26, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants