Running evals with external applications?

[mikael-andersson91]

I am currently building a GitHub Actions workflow that includes running the uipath eval for agent evaluators.

The workflow includes the following commands:

  orchestrator_url="https://cloud.uipath.com/"
  orchestrator_url="${orchestrator_url%%/}"
  base_url="${orchestrator_url}/ingka/GlobalTest"
  
  uipath auth \
    --client-id "***" \
    --client-secret "***" \
    --base-url "${base_url}" \
    --scope "OR.Administration.Read OR.Assets OR.BackgroundTasks OR.Execution OR.Execution.Read OR.Folders OR.Folders.Read OR.Jobs OR.Jobs.Read OR.Machines OR.Machines.Read OR.Monitoring OR.Queues OR.Queues.Read OR.Robots OR.Robots.Read OR.Settings OR.Settings.Read OR.Tasks OR.TestDataQueues OR.TestSetExecutions OR.TestSets OR.TestSetSchedules OR.Users OR.Users.Read StudioWebS2S.Project.CRUD StudioWebS2S.Share.CRUD StudioWebS2S.Workflow.CRUD EnterpriseContextService"
  
  uipath init
  
  uipath eval --workers 4 --output-file eval-results.json

Resulting in the errors below:

Resolved 112 packages in 0.65ms
Downloading ruff (10.7MiB)
Downloading cryptography (4.2MiB)
Downloading pydantic-core (2.0MiB)
Downloading aiohttp (1.7MiB)
Downloading pygments (1.2MiB)
Downloading openai (1.0MiB)
Downloading zstandard (5.3MiB)
Downloading tiktoken (1.1MiB)
 Downloaded tiktoken
 Downloaded pydantic-core
 Downloaded zstandard
 Downloaded ruff
 Downloaded aiohttp
 Downloaded cryptography
 Downloaded pygments
 Downloaded openai
Prepared 109 packages in 818ms
Installed 109 packages in 97ms
 + aiohappyeyeballs==2.6.1
 + aiohttp==3.13.3
 + aiosignal==1.4.0
 + aiosqlite==0.21.0
 + annotated-types==0.7.0
 + anyio==4.12.1
 + applicationinsights==0.11.10
 + attrs==25.4.0
 + bidict==0.23.1
 + certifi==2026.1.4
 + cffi==2.0.0
 + charset-normalizer==3.4.4
 + click==8.3.1
 + coverage==7.13.2
 + cryptography==46.0.4
 + distro==1.9.0
 + frozenlist==1.8.0
 + h11==0.16.0
 + httpcore==1.0.9
 + httpx==0.28.1
 + httpx-sse==0.4.3
 + idna==3.11
 + importlib-metadata==8.7.1
 + iniconfig==2.3.0
 + jiter==0.12.0
 + jsonpatch==1.33
 + jsonpath-ng==1.7.0
 + jsonpointer==3.0.0
 + jsonschema==4.26.0
 + jsonschema-pydantic-converter==0.1.6
 + jsonschema-specifications==2025.9.1
 + langchain==1.2.7
 + langchain-core==1.2.7
 + langchain-mcp-adapters==0.2.1
 + langchain-openai==1.1.7
 + langgraph==1.0.7
 + langgraph-checkpoint==4.0.0
 + langgraph-checkpoint-sqlite==3.0.3
 + langgraph-prebuilt==1.0.7
 + langgraph-sdk==0.3.3
 + langsmith==0.6.6
 + markdown-it-py==4.0.0
 + mcp==1.24.0
 + mdurl==0.1.2
 + mermaid-builder==0.0.3
 + mockito==1.5.5
 + msgpack==1.1.2
 + multidict==6.7.1
 + openai==2.16.0
 + openinference-instrumentation==0.1.43
 + openinference-instrumentation-langchain==0.1.58
 + openinference-semantic-conventions==0.1.25
 + opentelemetry-api==1.39.1
 + opentelemetry-instrumentation==0.60b1
 + opentelemetry-sdk==1.39.1
 + opentelemetry-semantic-conventions==0.60b1
 + orjson==3.11.5
 + ormsgpack==1.12.2
 + packaging==25.0
 + pathlib==1.0.1
 + pluggy==1.6.0
 + ply==3.11
 + propcache==0.4.1
 + pycparser==3.0
 + pydantic==2.12.5
 + pydantic-core==2.41.5
 + pydantic-function-models==0.1.11
 + pydantic-settings==2.12.0
 + pygments==2.19.2
 + pyjwt==2.10.1
 + pysignalr==1.3.0
 + pytest==9.0.2
 + python-dotenv==1.2.1
 + python-engineio==4.13.0
 + python-multipart==0.0.22
 + python-socketio==5.16.0
 + pyyaml==6.0.3
 + referencing==0.37.0
 + regex==2026.1.15
 + requests==2.32.5
 + requests-toolbelt==1.0.0
 + rich==14.3.1
 + rpds-py==0.30.0
 + ruff==0.14.14
 + simple-websocket==1.1.0
 + sniffio==1.3.1
 + sqlite-vec==0.1.6
 + sse-starlette==3.2.0
 + starlette==0.52.1
 + tenacity==9.1.2
 + tiktoken==0.12.0
 + tqdm==4.67.1
 + truststore==0.10.4
 + typing-extensions==4.15.0
 + typing-inspection==0.4.2
 + uipath==2.6.17
 + uipath-core==0.2.1
 + uipath-langchain==0.5.7
 + uipath-runtime==0.6.2
 + urllib3==2.6.3
 + uuid-utils==0.14.0
 + uvicorn==0.40.0
 + websockets==15.0.1
 + wrapt==1.17.3
 + wsproto==1.3.2
 + xxhash==3.6.0
 + yarl==1.22.0
 + zipp==3.23.0
 + zstandard==0.25.0
⠋ Authenticating with UiPath ...✓  Authentication successful.
⠋ Initializing UiPath project ...'uipath.json' already exists, skipping.
'bindings.json' already exists, skipping.
✓  Created 'entry-points.json' file with 1 entrypoint(s).
✓  Created 1 mermaid diagram file(s).
✓  Updated: CLAUDE.md, CLI_REFERENCE.md, SDK_REFERENCE.md, AGENTS.md, REQUIRED_STRUCTURE.md
Traceback (most recent call last):
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_base_service.py", line 147, in request_async
    response.raise_for_status()
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://cloud.uipath.com/ingka/GlobalTest/orchestrator_/odata/Users/UiPath.Server.Configuration.OData.GetCurrentUserExtended?$expand=PersonalWorkspace'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/bin/uipath", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 167, in eval
    should_register_progress_reporter = setup_reporting_prereq(no_report)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 56, in setup_reporting_prereq
    folder_key = asyncio.run(get_personal_workspace_key_async())
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/_utils/_folders.py", line 21, in get_personal_workspace_key_async
    response = await _get_personal_workspace_info_internal_async()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/_utils/_folders.py", line 33, in _get_personal_workspace_info_internal_async
    response = await uipath.api_client.request_async(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_api_client.py", line 59, in request_async
    return await super().request_async(method, url, scoped=scoped, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_base_service.py", line 150, in request_async
    raise EnrichedException(e) from e
uipath.platform.errors._enriched_exception.EnrichedException: 
Request URL: https://cloud.uipath.com/ingka/GlobalTest/orchestrator_/odata/Users/UiPath.Server.Configuration.OData.GetCurrentUserExtended?$expand=PersonalWorkspace
HTTP Method: GET
Status Code: 404
Response Content: {"message":"User not found","errorCode":1002,"traceId":"00-41ff632bdc84a3067c1ce960547f3d69-bd0fbdbc7ad6223c-00"}
Error: Process completed with exit code 1.

What permissions do I need to add to the external application for this to work? I have provided the required environment variable to link this with a Studio Web project as well.

[radu-mocanu]

Hi @mikael-andersson91

when using external applications there is no user identity involved, thus the personal workspace details (needed for trace reporting) cannot be retrieved.

there are 2 possible fixes:

1. use the --no-report flag (downside: traces won t be visible in Studio Web)
   uipath eval --workers 4 --output-file eval-results.json --no-report

2. manually set the UIPATH_FOLDER_KEY env var

implementation ref:

uipath-python/src/uipath/_cli/cli_eval.py

Line 48 in a7c588f

def setup_reporting_prereq(no_report: bool) -> bool:

[mikael-andersson91]

I went with the approach of setting UIPATH_FOLDER_KEY manually here, and it's starting to look better!

Traceback (most recent call last):
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/bin/uipath", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 187, in eval
    eval_context.entrypoint = entrypoint or auto_discover_entrypoint()
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/eval/_helpers/helpers.py", line 38, in auto_discover_entrypoint
    raise ValueError(
ValueError: No entrypoints found in uipath.json. Add a 'functions' section to uipath.json

What changes do I need to make in uipath.json for my evaluators to actually be able to run? I have followed the instructions for adding a new evaluator, as instructed in the Create New Evaluator and Register Evaluator sections. But it doesn't seem to trigger whatever changes are required.

[radu-mocanu]

if this is a plain uipath agent then you need to define at least one function in the uipath.json as per the specs https://github.com/UiPath/uipath-python/blob/main/specs/uipath.spec.md#4-functions

[mikael-andersson91]

After tinkering a bit more with my workflow, I've noticed that there does not seem to be a valid way for an external application to send reports to Studio Web? I have tried three different variations here:

Not working: uipath eval --workers 4 --output-file eval-results.json - With UIPATH_PROJECT_ID variable

❌ Error occurred: 
Request URL: https://cloud.uipath.com/ingka/studio_/backend/api/Project/25003685-c076-4230-b912-53f807042747
HTTP Method: GET
Status Code: 403
Response Content: {"code":"20000","message":"You are not authorized to access the requested projects.","translatedMessage":null,"family":0}
Traceback (most recent call last):
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 297, in eval
    asyncio.run(execute_eval())
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 276, in execute_eval
    async with ResourceOverwritesContext(
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_utils/_bindings.py", line 131, in __aenter__
    overwrites = await self.get_overwrites_callable()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/_utils/_studio_project.py", line 553, in get_resource_overwrites
    solution_id = await self._get_solution_id()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/_utils/_studio_project.py", line 495, in _get_solution_id
    response = await self.uipath.api_client.request_async(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_api_client.py", line 59, in request_async
    return await super().request_async(method, url, scoped=scoped, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_base_service.py", line 150, in request_async
    raise EnrichedException(e) from e
uipath.platform.errors._enriched_exception.EnrichedException: 
Request URL: https://cloud.uipath.com/ingka/studio_/backend/api/Project/25003685-c076-4230-b912-53f807042747
HTTP Method: GET
Status Code: 403
Response Content: {"code":"20000","message":"You are not authorized to access the requested projects.","translatedMessage":null,"family":0}

Not working: uipath eval --workers 4 --output-file eval-results.json --no-report - With UIPATH_PROJECT_ID variable

⠋ Authenticating with UiPath ...✓  Authentication successful.
⠋ Initializing UiPath project ...'uipath.json' already exists, skipping.
'bindings.json' already exists, skipping.
✓  Created 'entry-points.json' file with 1 entrypoint(s).
✓  Created 1 mermaid diagram file(s).
✓  Updated: CLAUDE.md, CLI_REFERENCE.md, SDK_REFERENCE.md, AGENTS.md, REQUIRED_STRUCTURE.md
Auto-discovered entrypoint: agent (src/main.py)
Auto-discovered evaluation set: evaluations/eval-sets/dataset-2.json
HTTP Request: GET https://cloud.uipath.com/ingka/studio_/backend/api/Project/25003685-c076-4230-b912-53f807042747 "HTTP/1.1 403 Forbidden"
❌ Error occurred: 
Request URL: https://cloud.uipath.com/ingka/studio_/backend/api/Project/25003685-c076-4230-b912-53f807042747
HTTP Method: GET
Status Code: 403
Response Content: {"code":"20000","message":"You are not authorized to access the requested projects.","translatedMessage":null,"family":0}
Traceback (most recent call last):
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 297, in eval
    asyncio.run(execute_eval())
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/cli_eval.py", line 276, in execute_eval
    async with ResourceOverwritesContext(
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_utils/_bindings.py", line 131, in __aenter__
    overwrites = await self.get_overwrites_callable()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/_utils/_studio_project.py", line 553, in get_resource_overwrites
    solution_id = await self._get_solution_id()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/_cli/_utils/_studio_project.py", line 495, in _get_solution_id
    response = await self.uipath.api_client.request_async(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_api_client.py", line 59, in request_async
    return await super().request_async(method, url, scoped=scoped, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/rpa-workflows/rpa-workflows/test-data/agent-sample/.venv/lib/python3.11/site-packages/uipath/platform/common/_base_service.py", line 150, in request_async
    raise EnrichedException(e) from e
uipath.platform.errors._enriched_exception.EnrichedException: 
Request URL: https://cloud.uipath.com/ingka/studio_/backend/api/Project/25003685-c076-4230-b912-53f807042747
HTTP Method: GET
Status Code: 403
Response Content: {"code":"20000","message":"You are not authorized to access the requested projects.","translatedMessage":null,"family":0}

Working version uipath eval --workers 4 --output-file eval-results.json --no-report - Remove UIPATH_PROJECT env var

Execution of evaluator works, but Studio Web connection is lost.

Conclusion

Since external applications cannot be added to Studio Web projects, it seems there is no way to utilize this connection in a GitHub Actions pipeline (no, I will not add a PAT for our pipeline, even if our org did have the option for it). Is this assessment correct, or am I missing something?

[mikael-andersson91]

@radu-mocanu, just checking in to see if I understood the compatibility/lack there of with Studio Web projects when running uipath eval with external application authentication