Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Short version: uipath add evaluator my-check creates evaluations/evaluators/custom/my_check.py from the shipped template, and uipath register evaluator my_check.py then fails: Error loading class: 'custom_evaluator' and Could not load class MyCheckEvaluator. The loader executes the file under the module name custom_evaluator without adding that module to sys.modules, and pydantic's generic subscription in the template (BaseEvaluatorConfig[MyCheckEvaluationCriteria]) looks the module up there. Registering the module in sys.modules before exec_module lets the class load (shown with a standalone load, not a full register run).
Evidence
- Versions:
uipath2.15.5 (latest),uip1.208.0-dev.10025, pydantic as installed byuipath; the CLI run used Python 3.14, the standalone load Python 3.13. uip codedagent add evaluator my-checkcreatesevaluations/evaluators/custom/my_check.py(its first message saysevals/evaluators/custom, the folder created isevaluations/..., andregisteralso reportsevals/...).- Without editing the file,
uip codedagent register evaluator my_check.py:Found custom evaluator file: evals/evaluators/custom/my_check.py Found custom evaluator class: MyCheckEvaluator Error loading class: 'custom_evaluator' Could not load class MyCheckEvaluator from my_check.py - The same load outside the CLI shows the cause (
output/trace.txt):importlib.util.spec_from_file_location("custom_evaluator", path)thenexec_moduleraisesKeyError: 'custom_evaluator'frompydantic/_internal/_generics.py(sys.modules[module].__dict__) atclass MyCheckEvaluatorConfig(BaseEvaluatorConfig[MyCheckEvaluationCriteria]). Aftersys.modules["custom_evaluator"] = modulethe class loads.
Code path
UiPath/uipath-pythonpackages/uipath/src/uipath/eval/evaluators/registration.py:66-86(bbd3f2b):load_evaluator_classbuilds the module withmodule_from_specand callsexec_modulewithoutsys.modules[...] = module; the broadexceptlogs theKeyErrortext only.packages/uipath/src/uipath/_cli/_templates/custom_evaluator.py.template: the generic subscription that triggers it.
Fix
- Set
sys.modules["custom_evaluator"] = modulebeforeexec_moduleand remove it in thefinally. - Log the exception with its type, so the message does not read as a missing class.
Repro files
repro/load_evaluator.py
import importlib.util, sys, traceback
from pathlib import Path
p = Path("evaluations/evaluators/custom/my_check.py").resolve()
spec = importlib.util.spec_from_file_location("custom_evaluator", p)
m = importlib.util.module_from_spec(spec)
try:
spec.loader.exec_module(m)
print("ok without sys.modules")
except Exception:
traceback.print_exc(limit=-6)
sys.modules["custom_evaluator"] = m
spec2 = importlib.util.spec_from_file_location("custom_evaluator", p)
m2 = importlib.util.module_from_spec(spec2); sys.modules["custom_evaluator"] = m2
spec2.loader.exec_module(m2)
print("ok with sys.modules entry; classes:", [n for n in dir(m2) if n.endswith("Evaluator")])output/register.txt
Found custom evaluator file: evals/evaluators/custom/my_check.py
Found custom evaluator class: MyCheckEvaluator
Error loading class: 'custom_evaluator'
? Could not load class MyCheckEvaluator from my_check.py
output/trace.txt
Traceback (most recent call last):
File "<SCRIPT>/load_evaluator.py", line 7, in <module>
spec.loader.exec_module(m)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "<frozen importlib._bootstrap_external>", line 759, in exec_module
File "<frozen importlib._bootstrap>", line 491, in _call_with_frames_removed
File "<PROBE_DIR>\a9\evaluations\evaluators\custom\my_check.py", line 13, in <module>
class MyCheckEvaluatorConfig(BaseEvaluatorConfig[MyCheckEvaluationCriteria]):
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<PYTHON_DIR>\Lib\site-packages\pydantic\main.py", line 1030, in __class_getitem__
submodel = _generics.create_generic_submodel(model_name, origin, args, params)
File "<PYTHON_DIR>\Lib\site-packages\pydantic\_internal\_generics.py", line 159, in create_generic_submodel
reference_module_globals = sys.modules[module].__dict__
~~~~~~~~~~~^^^^^^^^
KeyError: 'custom_evaluator'
ok with sys.modules entry; classes: ['BaseEvaluator', 'MyCheckEvaluator']
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity