Skip to content

uipath register evaluator: the evaluator scaffolded by uipath add evaluator fails with "Error loading class: 'custom_evaluator'" #2012

Open

Description

@danmunteanu1

Short version: uipath add evaluator my-check creates evaluations/evaluators/custom/my_check.py from the shipped template, and uipath register evaluator my_check.py then fails: Error loading class: 'custom_evaluator' and Could not load class MyCheckEvaluator. The loader executes the file under the module name custom_evaluator without adding that module to sys.modules, and pydantic's generic subscription in the template (BaseEvaluatorConfig[MyCheckEvaluationCriteria]) looks the module up there. Registering the module in sys.modules before exec_module lets the class load (shown with a standalone load, not a full register run).

Evidence

  1. Versions: uipath 2.15.5 (latest), uip 1.208.0-dev.10025, pydantic as installed by uipath; the CLI run used Python 3.14, the standalone load Python 3.13.
  2. uip codedagent add evaluator my-check creates evaluations/evaluators/custom/my_check.py (its first message says evals/evaluators/custom, the folder created is evaluations/..., and register also reports evals/...).
  3. Without editing the file, uip codedagent register evaluator my_check.py:
    Found custom evaluator file: evals/evaluators/custom/my_check.py
    Found custom evaluator class: MyCheckEvaluator
    Error loading class: 'custom_evaluator'
    Could not load class MyCheckEvaluator from my_check.py
    
  4. The same load outside the CLI shows the cause (output/trace.txt): importlib.util.spec_from_file_location("custom_evaluator", path) then exec_module raises KeyError: 'custom_evaluator' from pydantic/_internal/_generics.py (sys.modules[module].__dict__) at class MyCheckEvaluatorConfig(BaseEvaluatorConfig[MyCheckEvaluationCriteria]). After sys.modules["custom_evaluator"] = module the class loads.

Code path

  • UiPath/uipath-python packages/uipath/src/uipath/eval/evaluators/registration.py:66-86 (bbd3f2b): load_evaluator_class builds the module with module_from_spec and calls exec_module without sys.modules[...] = module; the broad except logs the KeyError text only.
  • packages/uipath/src/uipath/_cli/_templates/custom_evaluator.py.template: the generic subscription that triggers it.

Fix

  • Set sys.modules["custom_evaluator"] = module before exec_module and remove it in the finally.
  • Log the exception with its type, so the message does not read as a missing class.
Repro files

repro/load_evaluator.py

import importlib.util, sys, traceback
from pathlib import Path
p = Path("evaluations/evaluators/custom/my_check.py").resolve()
spec = importlib.util.spec_from_file_location("custom_evaluator", p)
m = importlib.util.module_from_spec(spec)
try:
    spec.loader.exec_module(m)
    print("ok without sys.modules")
except Exception:
    traceback.print_exc(limit=-6)
sys.modules["custom_evaluator"] = m
spec2 = importlib.util.spec_from_file_location("custom_evaluator", p)
m2 = importlib.util.module_from_spec(spec2); sys.modules["custom_evaluator"] = m2
spec2.loader.exec_module(m2)
print("ok with sys.modules entry; classes:", [n for n in dir(m2) if n.endswith("Evaluator")])

output/register.txt

Found custom evaluator file: evals/evaluators/custom/my_check.py
Found custom evaluator class: MyCheckEvaluator
Error loading class: 'custom_evaluator'
? Could not load class MyCheckEvaluator from my_check.py

output/trace.txt

Traceback (most recent call last):
  File "<SCRIPT>/load_evaluator.py", line 7, in <module>
    spec.loader.exec_module(m)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "<frozen importlib._bootstrap_external>", line 759, in exec_module
  File "<frozen importlib._bootstrap>", line 491, in _call_with_frames_removed
  File "<PROBE_DIR>\a9\evaluations\evaluators\custom\my_check.py", line 13, in <module>
    class MyCheckEvaluatorConfig(BaseEvaluatorConfig[MyCheckEvaluationCriteria]):
                                 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<PYTHON_DIR>\Lib\site-packages\pydantic\main.py", line 1030, in __class_getitem__
    submodel = _generics.create_generic_submodel(model_name, origin, args, params)
  File "<PYTHON_DIR>\Lib\site-packages\pydantic\_internal\_generics.py", line 159, in create_generic_submodel
    reference_module_globals = sys.modules[module].__dict__
                               ~~~~~~~~~~~^^^^^^^^
KeyError: 'custom_evaluator'
ok with sys.modules entry; classes: ['BaseEvaluator', 'MyCheckEvaluator']

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions