feat: add examples and tooling tests to run_tests_with_ollama_and_vllm

[jakelorocco]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

• Link to Issue: Fixes N/A

Adds the ability for the run_tests_with_ollama_vllm.sh script to specify whether to run the examples and tooling tests. Doesn't change the default behavior.

Output:
with examples

(mellea) ~/code/mellea4 ✓ % export WITH_EXAMPLES=1
(mellea) ~/code/mellea4 ✓ % ./test/scripts/run_tests_with_ollama_and_vllm.sh
[14:06:36] WARNING: CACHE_DIR not set. Ollama models will download to ~/.ollama (default)
[14:06:36] Using standalone log directory: logs/2026-04-10-14:06:36
[14:06:37] Ollama already running on 127.0.0.1:11434 — using existing server
[14:06:37] Pulling granite4:micro ...
success
[14:06:38] Model granite4:micro-h already pulled
[14:06:38] Pulling granite3.2-vision ...
success
[14:06:38] All ollama models ready.
[14:06:38] Warming up models...
[14:06:38]   Warming granite4:micro ...
[14:06:41]   Warming granite4:micro-h ...
[14:06:43]   Warming granite3.2-vision ...
[14:06:45] Warmup complete.
[14:06:45] vLLM disabled (WITH_VLLM=0). Pass WITH_VLLM=1 to enable, or run on a CUDA host for auto-detection.
[14:54:19] Tooling tests disabled (WITH_TOOLING_TESTS=0). Pass WITH_TOOLING_TESTS=1 to include test/tooling/.
[14:06:45] Starting pytest...
[14:06:45] Log directory: logs/2026-04-10-14:06:36
[14:06:45] Pytest args: --group-by-backend
============================= test session starts ==============================
platform darwin -- Python 3.12.13, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/jake/code/mellea4
configfile: pyproject.toml
plugins: nbmake-1.5.5, recording-0.13.4, cov-7.1.0, xdist-3.8.0, json-report-1.5.0, timeout-2.4.0, metadata-3.1.1, asyncio-1.3.0, Faker-40.12.0, langsmith-0.7.24, anyio-4.13.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 1214 items / 22 deselected / 1192 selected

docs/examples/aLora/101_example.py .                                     [  0%]
docs/examples/intrinsics/answerability.py .                              [  0%]
...
= 15 failed, 1152 passed, 20 skipped, 22 deselected, 2 xfailed, 3 xpassed, 130 warnings in 2356.23s (0:39:16) =

without examples:

(mellea) ~/code/mellea4 ✓ % ./test/scripts/run_tests_with_ollama_and_vllm.sh
[14:12:25] WARNING: CACHE_DIR not set. Ollama models will download to ~/.ollama (default)
[14:12:25] Using standalone log directory: logs/2026-04-10-14:12:25
[14:12:25] Ollama already running on 127.0.0.1:11434 — using existing server
[14:12:25] Pulling granite4:micro ...
success
[14:12:26] Model granite4:micro-h already pulled
[14:12:26] Pulling granite3.2-vision ...
success
[14:12:27] All ollama models ready.
[14:12:27] Warming up models...
[14:12:27]   Warming granite4:micro ...
[14:12:30]   Warming granite4:micro-h ...
[14:12:32]   Warming granite3.2-vision ...
[14:12:44] Warmup complete.
[14:12:44] vLLM disabled (WITH_VLLM=0). Pass WITH_VLLM=1 to enable, or run on a CUDA host for auto-detection.
[14:12:44] Examples disabled (WITH_EXAMPLES=0). Pass WITH_EXAMPLES=1 to include docs/examples/.
[14:54:19] Tooling tests disabled (WITH_TOOLING_TESTS=0). Pass WITH_TOOLING_TESTS=1 to include test/tooling/.
[14:12:44] Starting pytest...
[14:12:44] Log directory: logs/2026-04-10-14:12:25
[14:12:44] Pytest args: --group-by-backend
============================= test session starts ==============================
platform darwin -- Python 3.12.13, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/jake/code/mellea4
configfile: pyproject.toml
plugins: nbmake-1.5.5, recording-0.13.4, cov-7.1.0, xdist-3.8.0, json-report-1.5.0, timeout-2.4.0, metadata-3.1.1, asyncio-1.3.0, Faker-40.12.0, langsmith-0.7.24, anyio-4.13.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 1093 items / 18 deselected / 1075 selected

test/backends/test_huggingface.py

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code as added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[jakelorocco]

@avinash2692, please let me know if you agree / disagree with this. If you are good, I will add the flag to the script that actually runs the nightlies so that examples run (will leave tooling disabled for now).