refactor(DecoupleLoadFromRuntime): decouple eval load from eval runtime #1244
Conversation
github-actions
bot
added
test:uipath-langchain
| return eval_set, resolved_path | ||
|
|
||
| @staticmethod | ||
| async def load_evaluators( |
There was a problem hiding this comment.
This is moved from eval runtime as it needs to be adjacent to load eval set.
| return True | ||
|
|
||
|
|
||
| def _find_agent_model_in_runtime(runtime: UiPathRuntimeProtocol) -> str | None: |
There was a problem hiding this comment.
Moved from eval runtime
| f"eval_set_run_id={context.eval_set_run_id}" | ||
| ) | ||
| self.execution_id = ( | ||
| context.job_id or context.eval_set_run_id or str(uuid.uuid4()) |
There was a problem hiding this comment.
we need this for a consistent job id before and after suspend resume.
There was a problem hiding this comment.
Yes. This logic has been moved to _cli_eval.py right? Am I missing something?
Chibionos
left a comment
Chibionos
left a comment
There was a problem hiding this comment.
execution id being set to job key is important for the suspend resume to recover properly.
| The model name if found, None otherwise. | ||
| """ | ||
| # Check if this runtime implements the protocol | ||
| if isinstance(runtime, LLMAgentRuntimeProtocol): |
There was a problem hiding this comment.
We don't need this fallback anymore, the low-code runtime has the model in get_schema() (we can remove the LLMAgentRuntimeProtocol as well)
There was a problem hiding this comment.
Can I remove it in a separate small PR? I want this one to just focus on the decoupling,,
Chibionos
left a comment
Chibionos
left a comment
There was a problem hiding this comment.
approved, please ensure the suspend resume sample works properly with your change by running the tools suspend resume sample in langchain repo.
This allows customers to run evals dynamically without having to materialize to files.
akshaylive
force-pushed
the
akshaya/decouple_eval_load_from_runtime
branch
from
d83b481 to
2621a0e
Compare
|
Before: After: |
3 participants
This allows customers to run evals dynamically without having to materialize to files.
Development Package
uipath pack --nolockto get the latest dev build from this PR (requires version range).