predicates.py Apple Silicon VRAM estimate uses static heuristic instead of actual available memory, causing GPU-gated tests to run on under-resourced hardware

[ajbozarth]

_apple_silicon_vram_gb() in test/predicates.py estimates usable GPU memory using a static formula:

min(total * 0.75, total - 16)                                                                                                                                                                                                        

On a 32 GB machine this gives 16 GB, but psutil.virtual_memory().available at runtime shows ~18.5 GB actually free. More importantly, the formula is evaluated at pytest collection time with no knowledge of current system load, and
it doesn't align with how CUDA reports free VRAM (which the audit-markers skill's formula was written against).

Observed effect: test/stdlib/components/intrinsic/test_core.py has require_gpu(min_vram_gb=12), which the skill correctly computed using the adapter accumulation formula. However, on a 32 GB Apple Silicon machine the
predicate reports 16 GB available, so the gate passes. The model then runs under memory pressure and produces incoherent output (garbled JSON with raw token IDs), failing the test.

Proposed fix

Replace the static heuristic in _apple_silicon_vram_gb() with psutil.virtual_memory().available, which reflects actual free memory at collection time and is consistent with how CUDA reports free VRAM. The audit-markers skill
formula would then work correctly on both platforms without changes.

Branch: fix/audit-markers-vram-cross-check

Notes

• psutil is already a project dependency (used by require_ram() in the same file)
• The CUDA path in _gpu_vram_gb() reports total device memory, not free — may also be worth revisiting for consistency, but is a separate concern

[ajbozarth]

@planetf1 I hit this issue while testing #782 and tried working with Claude to fix it, but hit EOD right before leaving for my 4-day weekend vacation. I figured you could take a look once you're back.

I'm not sure if Claudes description above makes sense, my tl;dr is that silicon uses unified ram and has to be treated differently than cuda so the test properly marked 12GB was pushing my ram to pressure and failing, this implies that a 12GB tests should not run on a 32GB Mac?

So I just had Claude summarize our findings and open this issue

[planetf1]

I did consider this when I originally implemented -- but opted for a 'finger in the air' heuristic which mapped to my personal experience as a first pass.

We use min(total * 0.75, total - 16)

The 0.75 factor is an approximation of the max GPU allocation for unified memory (see sysctl iogpu.wired_limit_mb to view/change), and the 16GB free space for IDEs, and other desktop tools

So in the case of a 32GB machine we went up with min(24,16) = 16
For 64GB we have min(48,48) = 48

Both seem reasonable

You said you 'hit' this issue - do you mean you felt your machine was under-utilized? Or overloaded? or did tests fail? Or was it a code review observation?

I was indeed aware of the tool to get actual virtual memory usage - the problem is that it too is imperfect - it's the amount of memory available in the perfect world - sometimes pages are slow to release. It also includes virtual memory and we have to be careful not to introduce paging.

it also makes tests more non-deterministic since what tests run would fluctuate by more than a fairly fixed set based on machine config

So it could be worse, not better

A more sophisticated approach might therefore also need to consider

• memory pressure (as reported by sys monitor), paging activity

Even better would be, via whatever stack/tool is in use, to discover whether all the model layers are loaded onto the gpu, or we missed

[planetf1]

You said:

Observed effect: test/stdlib/components/intrinsic/test_core.py has require_gpu(min_vram_gb=12), which the skill correctly computed using the adapter accumulation formula. However, on a 32 GB Apple Silicon machine the
predicate reports 16 GB available, so the gate passes. The model then runs under memory pressure and produces incoherent output (garbled JSON with raw token IDs), failing the test.

Were you able to reproduce this multiple times?
Do you know what you memory usage (especially pinned memory) was?
How about memory pressure?
Can we get any debugging from the framework (may be worth trying to add this to better understand what is going on) - I would expect calls to pass, fail.. garbled output is a little concerning

Another thing to consider -- there's two parts to this
a) How much gpu memory we consider is 'usable' (the heuristic)
b) how much gpu memory we guess (with the agent) that the test might use

b) should also be considered - I think we can only hope that the agent has a good initial guess, but that tweaking further may be needed for some tests

[ajbozarth]

memory pressure

this is the limiting factor I hit. I would watch the test max out memory pressure in the system console during that test and it would always fail.

Were you able to reproduce this multiple times?

yes this particulate test (in desc) was consistently failing

[planetf1]

Do you have any idea how much memory the test was trying to use? ie was the estimation assigned in the test appropriate
(will try this out myself when I get back to this issue tomorrow etc)

[ajbozarth]

Do you have any idea how much memory the test was trying to use?

I'm not actually sure how to check the exit mem usage, I only have the activity monitor with doesn't show that. Is there a tool/app on macOS you know of that I could use instead?

[planetf1]

I was just thinking about what activity monitor shows - there are multiple memory columns that can be viewed, as well as a summary which includes 'wired' memory (I may have mentioned pinned).

when I look at this PR I'll figure it out...

[planetf1]

Ran into this whilst running the full docs/examples/ suite. The SOFAI graph-colouring example (sofai_graph_coloring.py) loads two models simultaneously — phi:2.7b (1.6 GB) and pielee/qwen3-4b-thinking-2507_q8 (4.3 GB). On a 32 GB Apple Silicon machine with other models still resident from earlier tests (guardian 8b at ~11 GB, llama3.2 at ~5.6 GB), ollama runs out of headroom and the model runner crashes:

ollama._types.ResponseError: model runner has unexpectedly stopped,
this may be due to resource limitations or an internal error (status code: 500)

Notably, the example has no require_gpu() or require_ram() gate, so even a corrected VRAM estimate wouldn't skip it — it just runs and dies under memory pressure.

Two things would help here:

1. The predicate fix proposed in this issue (actual available memory, not static heuristic)
2. Adding a resource gate to the SOFAI example so it skips gracefully when there isn't enough headroom

[ajbozarth]

output for ref:

Terminal output

$ uv run pytest
=========================================================================================================== test session starts ===========================================================================================================
platform darwin -- Python 3.12.8, pytest-9.0.0, pluggy-1.6.0
rootdir: /Users/ajbozarth/workspace/ai/mellea
configfile: pyproject.toml
testpaths: test, docs
plugins: nbmake-1.5.5, recording-0.13.4, anyio-4.11.0, xdist-3.8.0, json-report-1.5.0, timeout-2.4.0, metadata-3.1.1, asyncio-1.3.0, langsmith-0.6.6, Faker-37.12.0, cov-7.0.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 890 items / 19 deselected / 2 skipped / 871 selected                                                                                                                                                                            

test/backends/test_adapters/test_adapter.py .                                                                                                                                                                                       [  0%]
test/backends/test_bedrock.py s                                                                                                                                                                                                     [  0%]
test/backends/test_huggingface.py sssssssssssssssssss                                                                                                                                                                               [  2%]
test/backends/test_huggingface_tools.py s                                                                                                                                                                                           [  2%]
test/backends/test_litellm_ollama.py .......F                                                                                                                                                                                       [  3%]
test/backends/test_litellm_watsonx.py ssss                                                                                                                                                                                          [  3%]
test/backends/test_mellea_tool.py ........                                                                                                                                                                                          [  4%]
test/backends/test_model_options.py .....                                                                                                                                                                                           [  5%]
test/backends/test_ollama.py .....X.F..                                                                                                                                                                                             [  6%]
test/backends/test_openai_ollama.py .......F.....                                                                                                                                                                                   [  8%]
test/backends/test_openai_vllm.py sssssss                                                                                                                                                                                           [  8%]
test/backends/test_tool_calls.py ...                                                                                                                                                                                                [  9%]
test/backends/test_tool_decorator.py ...................                                                                                                                                                                            [ 11%]
test/backends/test_tool_helpers.py ...                                                                                                                                                                                              [ 11%]
test/backends/test_tool_validation_integration.py .................................                                                                                                                                                 [ 15%]
test/backends/test_vision_ollama.py ....                                                                                                                                                                                            [ 15%]
test/backends/test_vision_openai.py ....                                                                                                                                                                                            [ 16%]
test/backends/test_watsonx.py sssssssssss                                                                                                                                                                                           [ 17%]
test/cli/test_alora_train.py ....                                                                                                                                                                                                   [ 18%]
test/cli/test_alora_train_integration.py ss                                                                                                                                                                                         [ 18%]
test/core/test_astream_exception_propagation.py .....                                                                                                                                                                               [ 18%]
test/core/test_astream_incremental.py ......                                                                                                                                                                                        [ 19%]
test/core/test_astream_mock.py ......                                                                                                                                                                                               [ 20%]
test/core/test_base.py ....                                                                                                                                                                                                         [ 20%]
test/core/test_component_typing.py ........                                                                                                                                                                                         [ 21%]
test/core/test_model_output_thunk.py ..                                                                                                                                                                                             [ 21%]
test/decompose/test_decompose.py ..........                                                                                                                                                                                         [ 23%]
test/formatters/granite/test_intrinsics_formatters.py ........................................................x.................                                                                                                    [ 31%]
test/formatters/test_template_formatter.py ................                                                                                                                                                                         [ 33%]
test/helpers/test_event_loop_helper.py ....                                                                                                                                                                                         [ 33%]
test/helpers/test_server_type.py ................                                                                                                                                                                                   [ 35%]
test/plugins/test_all_payloads.py ...................................................................................................                                                                                               [ 47%]
test/plugins/test_blocking.py ................                                                                                                                                                                                      [ 48%]
test/plugins/test_build_global_context.py .......                                                                                                                                                                                   [ 49%]
test/plugins/test_decorators.py .........                                                                                                                                                                                           [ 50%]
test/plugins/test_execution_modes.py ...........................                                                                                                                                                                    [ 53%]
test/plugins/test_hook_call_sites.py ..............................                                                                                                                                                                 [ 57%]
test/plugins/test_manager.py ss......                                                                                                                                                                                               [ 58%]
test/plugins/test_mellea_plugin.py .......                                                                                                                                                                                          [ 59%]
test/plugins/test_payloads.py ..........                                                                                                                                                                                            [ 60%]
test/plugins/test_pluginset.py .........                                                                                                                                                                                            [ 61%]
test/plugins/test_policies.py ......                                                                                                                                                                                                [ 61%]
test/plugins/test_policy_enforcement.py ..........                                                                                                                                                                                  [ 63%]
test/plugins/test_priority_ordering.py ..............                                                                                                                                                                               [ 64%]
test/plugins/test_scoping.py ...................................                                                                                                                                                                    [ 68%]
test/plugins/test_tool_hooks_redaction.py .......                                                                                                                                                                                   [ 69%]
test/plugins/test_unregister.py .........                                                                                                                                                                                           [ 70%]
test/stdlib/components/docs/test_document.py ...                                                                                                                                                                                    [ 70%]
test/stdlib/components/docs/test_richdocument.py .....s                                                                                                                                                                             [ 71%]
test/stdlib/components/intrinsic/test_core.py ..F                                                                                                                                                                                   [ 71%]
test/stdlib/components/intrinsic/test_guardian.py ......                                                                                                                                                                            [ 72%]
test/stdlib/components/intrinsic/test_rag.py ....F..                                                                                                                                                                                [ 73%]
test/stdlib/components/test_chat.py .                                                                                                                                                                                               [ 73%]
test/stdlib/components/test_genslot.py ...................                                                                                                                                                                          [ 75%]
test/stdlib/components/test_hello_world.py ..                                                                                                                                                                                       [ 75%]
test/stdlib/components/test_mify.py ...........                                                                                                                                                                                     [ 77%]
test/stdlib/components/test_transform.py ..                                                                                                                                                                                         [ 77%]
test/stdlib/requirements/test_reqlib_markdown.py ......                                                                                                                                                                             [ 78%]
test/stdlib/requirements/test_reqlib_python.py .............sss.....                                                                                                                                                                [ 80%]
test/stdlib/requirements/test_reqlib_tools.py .                                                                                                                                                                                     [ 80%]
test/stdlib/requirements/test_requirement.py .....                                                                                                                                                                                  [ 81%]
test/stdlib/sampling/test_majority_voting.py ..                                                                                                                                                                                     [ 81%]
test/stdlib/sampling/test_sampling_ctx.py ..                                                                                                                                                                                        [ 81%]
test/stdlib/sampling/test_sofai_graph_coloring.py .........................                                                                                                                                                         [ 84%]
test/stdlib/sampling/test_sofai_sampling.py .....................                                                                                                                                                                   [ 86%]
test/stdlib/sampling/test_think_budget_forcing.py ..                                                                                                                                                                                [ 87%]
test/stdlib/test_base_context.py .....                                                                                                                                                                                              [ 87%]
test/stdlib/test_chat_view.py ..                                                                                                                                                                                                    [ 87%]
test/stdlib/test_functional.py ....                                                                                                                                                                                                 [ 88%]
test/stdlib/test_session.py s.......                                                                                                                                                                                                [ 89%]
test/stdlib/test_spans.py .x                                                                                                                                                                                                        [ 89%]
test/telemetry/test_logging.py ........                                                                                                                                                                                             [ 90%]
test/telemetry/test_metrics.py .......................................                                                                                                                                                              [ 94%]
test/telemetry/test_metrics_backend.py ....s....                                                                                                                                                                                    [ 95%]
test/telemetry/test_metrics_latency.py ...                                                                                                                                                                                          [ 96%]
test/telemetry/test_metrics_plugins.py ........                                                                                                                                                                                     [ 97%]
test/telemetry/test_metrics_token.py ....                                                                                                                                                                                           [ 97%]
test/telemetry/test_tracing.py ..............                                                                                                                                                                                       [ 99%]
test/telemetry/test_tracing_backend.py ssssss                                                                                                                                                                                       [100%]

================================================================================================================ FAILURES =================================================================================================================
____________________________________________________________________________________________________________ test_async_avalue ____________________________________________________________________________________________________________

session = <mellea.stdlib.session.MelleaSession object at 0x140c51b80>

    async def test_async_avalue(session):
        mot1, _ = await session.backend.generate_from_context(
            CBlock("Say Hello."), SimpleContext()
        )
        m1_final_val = await mot1.avalue()
        assert m1_final_val is not None
        assert m1_final_val == mot1.value
    
        # Verify telemetry fields are populated
        assert mot1.usage is not None
        assert mot1.usage["prompt_tokens"] >= 0
        assert mot1.usage["completion_tokens"] > 0
        assert mot1.usage["total_tokens"] > 0
        assert isinstance(mot1.model, str)
        assert mot1.provider == "litellm"
        assert mot1.streaming is False
>       assert mot1.ttfb_ms is None
E       assert 503.11499999999995 is None
E        +  where 503.11499999999995 = ModelOutputThunk(Hello! How may I assist you today?).ttfb_ms

test/backends/test_litellm_ollama.py:218: AssertionError
____________________________________________________________________________________________________________ test_async_avalue ____________________________________________________________________________________________________________

session = <mellea.stdlib.session.MelleaSession object at 0x140f61880>

    async def test_async_avalue(session) -> None:
        mot1, _ = await session.backend.generate_from_context(
            CBlock("Say Hello."), SimpleContext()
        )
        m1_final_val = await mot1.avalue()
        assert m1_final_val is not None
        assert m1_final_val == mot1.value
    
        # Verify telemetry fields are populated
        assert mot1.usage is not None
        assert mot1.usage["prompt_tokens"] >= 0
        assert mot1.usage["completion_tokens"] > 0
        assert mot1.usage["total_tokens"] > 0
        assert isinstance(mot1.model, str)
        assert mot1.provider == "ollama"
        assert mot1.streaming is False
>       assert mot1.ttfb_ms is None
E       assert 260.718 is None
E        +  where 260.718 = ModelOutputThunk(Hello! How may I assist you today?).ttfb_ms

test/backends/test_ollama.py:211: AssertionError
---------------------------------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------------------------------
=== 15:34:12-INFO ======
Starting Mellea session: backend=ollama, model=granite4:micro, context=SimpleContext
----------------------------------------------------------------------------------------------------------- Captured log setup ------------------------------------------------------------------------------------------------------------
INFO     fancy_logger:session.py:246 Starting Mellea session: backend=ollama, model=granite4:micro, context=SimpleContext
____________________________________________________________________________________________________________ test_async_avalue ____________________________________________________________________________________________________________

m_session = <mellea.stdlib.session.MelleaSession object at 0x1415732f0>

    async def test_async_avalue(m_session) -> None:
        mot1, _ = await m_session.backend.generate_from_context(
            CBlock("Say Hello."), SimpleContext()
        )
        m1_final_val = await mot1.avalue()
        assert m1_final_val is not None
        assert m1_final_val == mot1.value
    
        # Verify telemetry fields are populated
        assert mot1.usage is not None
        assert mot1.usage["prompt_tokens"] >= 0
        assert mot1.usage["completion_tokens"] > 0
        assert mot1.usage["total_tokens"] > 0
        assert isinstance(mot1.model, str)
        assert mot1.provider == "openai"
        assert mot1.streaming is False
>       assert mot1.ttfb_ms is None
E       assert 336.167 is None
E        +  where 336.167 = ModelOutputThunk(Hello! How can I assist you today?).ttfb_ms

test/backends/test_openai_ollama.py:212: AssertionError
_____________________________________________________________________________________________________ test_find_context_attributions ______________________________________________________________________________________________________

mot = ModelOutputThunk(None)
chunk = ChatCompletionResponse(choices=[ChatCompletionResponseChoice(index=0, message=AssistantMessage(content='[ \t\t\t\t\t\t...ole='assistant', tool_calls=None, reasoning_content=None), logprobs=None, finish_reason='stop')], prompt_logprobs=None)
rewritten = ChatCompletion(messages=[UserMessage(content='<c70> Who were the members of The Metal Ono Band, which was formed by Yo...$defs/_MODEL_OUTPUT_ENTRY'}, 'title': '_MODEL_OUTPUT', 'type': 'array'}}), temperature=0.0, max_completion_tokens=4096)
result_processor = <mellea.formatters.granite.intrinsics.output.IntrinsicsResultProcessor object at 0x150d20e90>, input_ids = tensor([[100264,   9125, 100265,  ..., 100264,  78191, 100265]],
       device='mps:0')

    async def granite_formatters_processing(
        mot: ModelOutputThunk,
        chunk: granite_formatters.ChatCompletionResponse,
        rewritten: granite_formatters.ChatCompletion,
        result_processor: granite_formatters.IntrinsicsResultProcessor,
        input_ids,
    ):
        try:
>           res = result_processor.transform(chunk, rewritten)  # type: ignore
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

mellea/backends/huggingface.py:581: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mellea/formatters/granite/base/io.py:182: in transform
    return self._transform_impl(chat_completion_response, chat_completion)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mellea/formatters/granite/intrinsics/output.py:1267: in _transform_impl
    self._transform_choice(c, chat_completion)
mellea/formatters/granite/intrinsics/output.py:1305: in _transform_choice
    parsed_json = json.loads(content)
                  ^^^^^^^^^^^^^^^^^^^
../../../.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/json/decoder.py:338: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.decoder.JSONDecoder object at 0x1058c1700>
s = '[ \t\t\t\t\t\t\t\t\t\t\t\t              \n                           \n    \t             \t\t\t\t\t \n        \n    ...t,\n\n249   \n   ,  5      \n    \n    \t  \n\n   , 25\n\n\n\n\n\n\n\n\n\n\n\n\n\n      \n    \n  \n   , 207 \n    \n '
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
>           obj, end = self.scan_once(s, idx)
                       ^^^^^^^^^^^^^^^^^^^^^^
E           json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1332 column 2 (char 11827)

../../../.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/json/decoder.py:354: JSONDecodeError

The above exception was the direct cause of the following exception:

backend = <mellea.backends.huggingface.LocalHFBackend object at 0x150a48200>

    @pytest.mark.qualitative
    def test_find_context_attributions(backend):
        """Verify that the context-attribution intrinsic functions properly."""
        context, assistant_response, documents = _read_rag_input_json(
            "context-attribution.json"
        )
        expected = _read_rag_output_json("context-attribution.json")
    
>       result = core.find_context_attributions(
            assistant_response, documents, context, backend
        )

test/stdlib/components/intrinsic/test_core.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mellea/stdlib/components/intrinsic/core.py:90: in find_context_attributions
    result_json = call_intrinsic(
mellea/stdlib/components/intrinsic/_util.py:39: in call_intrinsic
    model_output_thunk, _ = mfuncs.act(
mellea/stdlib/functional.py:98: in act
    out = _run_async_in_thread(
mellea/helpers/event_loop_helper.py:105: in _run_async_in_thread
    return __event_loop_handler(co)
           ^^^^^^^^^^^^^^^^^^^^^^^^
mellea/helpers/event_loop_helper.py:77: in __call__
    return asyncio.run_coroutine_threadsafe(co, self._event_loop).result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py:456: in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
../../../.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
mellea/stdlib/functional.py:584: in aact
    await result.avalue()
mellea/core/base.py:410: in avalue
    await self.astream()
mellea/core/base.py:509: in astream
    await self._process(self, chunk)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

mot = ModelOutputThunk(None)
chunk = ChatCompletionResponse(choices=[ChatCompletionResponseChoice(index=0, message=AssistantMessage(content='[ \t\t\t\t\t\t...ole='assistant', tool_calls=None, reasoning_content=None), logprobs=None, finish_reason='stop')], prompt_logprobs=None)
rewritten = ChatCompletion(messages=[UserMessage(content='<c70> Who were the members of The Metal Ono Band, which was formed by Yo...$defs/_MODEL_OUTPUT_ENTRY'}, 'title': '_MODEL_OUTPUT', 'type': 'array'}}), temperature=0.0, max_completion_tokens=4096)
result_processor = <mellea.formatters.granite.intrinsics.output.IntrinsicsResultProcessor object at 0x150d20e90>, input_ids = tensor([[100264,   9125, 100265,  ..., 100264,  78191, 100265]],
       device='mps:0')

    async def granite_formatters_processing(
        mot: ModelOutputThunk,
        chunk: granite_formatters.ChatCompletionResponse,
        rewritten: granite_formatters.ChatCompletion,
        result_processor: granite_formatters.IntrinsicsResultProcessor,
        input_ids,
    ):
        try:
            res = result_processor.transform(chunk, rewritten)  # type: ignore
        except json.JSONDecodeError as e:
>           raise Exception(f"Intrinsic did not return a JSON: {chunk}") from e
E           Exception: Intrinsic did not return a JSON: choices=[ChatCompletionResponseChoice(index=0, message=AssistantMessage(content='[ \t\t\t\t\t\t\t\t\t\t\t\t              \n                           \n    \t             \t\t\t\t\t \n        \n        \n \n            \n \n\n\n\n    {\n                                                                                 \t\t\t    \t\n                        \n\t\t\t\t\t\t\t\t\t\t  \n                        \n\t\t\t\t\t \n\n    \n\n\n\n\n\n\n\n\n\n\n                      \n                    \n\n\t \t\t    \t\n  \n   "r"\n\n:255762762181448,\n\n\n\n\n\n\n\n\n\n\n \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t \n        \n\n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n                                                  \n "c":\n\t\t\t\t\t\t\t\t\t\t\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n   [ \t\t\t\t\t\t    \n\n\n\n\n\n\n\n\n\n\n\n\n\n       \n ]\n \n\n    \n\n\n  \n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n    \t  \n                                \t\t\t\t\n\n\n\n\n\n\n\n\t\t\n\t\n  \n \n     \t\t\t\t\t\n\t\t\t\t\t\n                                                                                \t\t\t\t\t\n\t\t\t\t\t\n  \t\t\t\t\t\n\t\t\t\t\t\n  \t}  \n\n\n\n\n\n\n\n\n\n\n \n\n,   \t\t\n\t\t\n\t\t\n\t  \t\t\t\t\t\t\t\t\t   { \t\t          "r":\n \n\n\n\n\n\n\n\n\n\n\n 23 \t\t        , "c" \t \t\t\t\t\t\t     \n\n\n\n\n\n\n\n \n\n  \n\t\t\t\n\t\t\t\n                                                                                                          \n: \n\n\n\n\n\n\n\n         \t \n        \n[]      \n   },{     \t\t\t             "r"\n    \n        \t   :\n\n\n\n  \t  29 ,                                    \n                                                                    "c"                            \n    \t\t                 \n:     [] },{\n  \n\n                                                                                    \t\t                          \n\n\n  \t\t          "r"        \t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n  \n\n    \t      \t\t\t\t\t\t\t\t\t\t\t\t\t\n                                             \t\t\t      \n  \n\n \n    \t   \n            \t\t :\n\n\n\n   \n  8\n,           \n        \n        \n\n\n\n\n\n\n\n     \t\t\t\t\t\n\t\t\t\t\t\n    \n    \n    \n    \n "c"     \n     \n     \t\t                                                                                                                                        \t\t\t\t       \n \n  \t  \n  \t                \t               \t    \t\t\t \t\t\t\t\t\t\n\n\n   \t\t\n\t\t\n\t\t\n \n   \n  \n   \n\n\n\n\n\n\n\n\n\n\n\n\n      \n    \t \n                                                                                                                                      \n        \n        \n:        \n\n    \n\n     \n  \t       \n\n\t\t\t\t           \n     \n\t\t\t \n    [10, 29,    350,\t\t\t \n   72, \t290 \n\n\n    ]}\n \n, \t \t {  \n\n\n\n\n\n   \n       "r":\t\t\t\t      1310     \n  \n     ,    \n           \n     \n                "c"  \n   \n          \t\t\t\t\t\t\t\t\t  \t \t\t    \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t   \n\n     \n\n               :  \n      \t\t\t\t\t \n       \t            \t\t           \n\n\n\n\n  \t                          \n                    \n     \t\t\t\t\t\t\t        \n        \n\n                                                                                \t\t\t        \t \t        \n                                                                         \t\t\t\t\t\t   []   \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t \n     \t\t                 }        \n  \n  \n                              \n\n\n\n,\t\t  \n       \n\n\n\n\n\n\n  {\n           \n        \n        \n\n  \t\t"r": 431        \t\t   \t,            \n      \n    "c"\n       \n   \t\t\t\t\t\t   \t     \t \n             \n    \t      :                                                                 \t\t\t\t\t\t\t          \n    \n \n\n \t[  4651    \t                                                        \t\t,  100,   619                                                                                                ,  3       \t    \t, 1   ,107155     , 0   \t     \t\t, 2       ]\n\n\n  \n       \t\t          \n  \n  \n      }, \t \n    \t  \t           \n\n\n\n\n\n\n\n\n\n\n      \t\n\t\t\t\t\n{  "r"            \t   \n  \t    \n                                                                                        \n        \n    \t    \t                      \t       \t\t \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n    \n              \n\n\n  \n        \t \n      \n\n      \t   \t      \n\n   \n                                                                                                                 :    1        ,    "c"    \n\n  \n                   \n       :    \t         \n\n    \n      [  10,                                 \n\n\n22 \t,  29 ]\n\n\n \t \t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t   \t   \t  \n                                     \n     \t     \n    \n    \n    \n  \t  \t  \t           \t      \t\t\n         \n     \n    \n      }, {      \t           \t                   \n    \n\n       "r"  \t        \n        \n        \n  \n\n :\n\n\n\n  \n       \n    \t200\n , \t   \n               "c"\n\t            \n        \n\t  \t                                                 \t        \n\n  \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n   \n\n\n\n\n\n   \t     \t \t\t\t\t\t\t      \t\t\t\t\t           \n  \n          \t\t\t\t  \n :\n\n\n\n    \n\n\n\n\n\n         \n    \n   \t  [  361  ]                                                            \n                             \n },\n  { \t\t "r"   \n\n\n\n         \n\n\n\n\n\n  \t\t                \t\t\t\n\t\t\n\t\t\n                                           \n                        \t \n        \n   \n    \t\t\t\t\t\t\t  \n\n                       \t\t\t\t\t\n    \n        \n                        \n\t             \t\n             \n\n\n     :697 ,     \t\t\t\t\t      \n\n\n\n\n\n\n\n\n\n\n\n       "c"   : \t\t    [] \n           \t\t\t\t\t\t\t \t    \t   \t\t\t\t\t\t  \n           }\n\n\t \t\t\t\t\t\n\t\t\t\t\t\n\t\t, \n\n\n\n   \n\n\n\n\t\t\t\t\t       \t          \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n  \n    {                                                            \n\n \n\n"r":           \t    155222  \n   ,\t\t\t\t\t\t                     \n                    \n                \n    \n    \n    \n\t                  \t\t\t     \n\n     \t     \n \n\n\t\t\t\t\n\t\t\t\t\n    \n     \n  \n \n  \n       \t        \n \n    \t \t\t\t\t    \t  \t\n      \n\n                                           "c"  \t\n       \t\t      : \n\t                    \n                    \n              \n                        \n    \t [\n\n     16                                                            ,             \n\n 28     , \n\n\n\n\n\n\n\n\n\n\n\n   14                 \t    \n    \n \n       ,751  \n \n\n   ,208  \n  \t\t\t\t\t\t    \n\n\n\n\n\n   \t  \n\n\n \n\n\n\n\n\n            \n     \n    , 1140071, 16 \n    ,926   \n          ,             10286\n  \t,      \t \t\t 13822   \n    , 80       \t\t                       \n\n, 303 , 926,  0\n      \n        \t\t             \n   \n   \n          \n     \t\t\t\t                \n\n\n\t, 980    ,\n\n 1 \t\t                   \t   \n \n,                                                                 326\n     \t\n ,\n\n\n   \n   8   \n   \n \t\t\t\t\t\t\t\n\n,  210   \n   \n,  14         \t , 9     \n  \n     \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n   \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n    \t\n     \t\t   \t\t\n\n\n,  19  \n\n\n\n\n\n    ,  0   \t                                                         \n\n         ,\t     8,  100310310 \n      \t\t   \n           \t\t\n\t\t\n\t\t\n     , 7  \t,407     \t \t\t\t\t\t\t\t\t\n  \t      ,  191             \t\t\t\t\t    \t,3    \n       \t                 \n    \n   \n    ,\t\t\t         \n\n  255255\n  \t, 129          \n    \t \n\n       \n\n                                                                    \n\n\n\n\n   \t\t\t\t\t\t\t\t  \t      \t  \n\n\n\n\n\n     \t  \n         \t         ,200  ,   \n\n   \t487,        8             \n            \t\t   \n       \t, 130\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t     \t\t\t\t\t         \n                                                                             \n\n                                                                      ,  \n   3         \t\t\n\t     ,     50  \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n   \n\n\n\n\n\n\n            \n          ,\n                                   57                                                  \n\n\n\n                ,     12   \t\t\t\t\t\t\t\t\t\t\t\t  \n  \n     \n   \n \t        \t \n\n\n        \n  \n          \n      \n\n                      \t   \n\t       \n  \n\n\t  \n                \n                \n\n \n      , 28  ,    13 ,0  \n,  10, 35\n \t\t\t\t\t\t             \t,  37  \n    \n \n          \n    \n       \n\n\n\n\n\n\n\n                                                       ,\n\n\n\n\n\n\n\n\n\n\n\n\n\n    1557             \t                                                                                            \n         \t  \n     \t       \n    \n,     \n230                                    ] \t                            \n\n              },  \n{    \t\n\n          \t \n\n  "r":262 \n\n \t  ,     \t\t\t\t\t\t   \t\t\t\t\t\t\t\t\t\t\t\t   \n    \n  "c":[137\t  \n    \n   , 7303941  \t  , 3  \n         \t      \t  \t \t\t, 31  ,  0,969         \t \n\t  \n  \n      ,\n\n\n123   \t\t\t\t\t\t\t\t \t       \t\t\t\t\t\t\t\t \t, 3026970343    \n  \n     \t\n  \t  ,503\n      \t           \n   \t\t\t\t                 \t      \n , 265  \n\n\n\n\n      , 9988577 \t           \n                     \n                    \n  \n    \n            \n        \n        \n          \n  ,  583 \n    \t, 13      \t      \t   \t\t\t\t\n\n\t  \t ,  180   \t  ]\n  \t\t             \t    \n        \n         \t}    \t\t\t\t     \n      \n ,\t             {      \t\t     \t    \t"r"\n         \n   :\n\n\n\n\n\n14   \t    \t, "c"  \n    \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t                                \n   \n\n                   \t\t  \n           \n\t   \t     \n \t\t\t\t\t\t  \t    : \n                  \n    \t  \n              \n  \n   \t          \n\n                                                                         \n\n \n       \t\t       \t\t[   \n  \n \t19    \t\t  \n                                                                      \n                   , 160      \n\n\n   \n     \t \t \t\n    \t  \t\t\t\t\t\n  , \n  297   ,\n\n28  \n       ,\n\n       212  \t, 18, 7 \t                      \n   , 1, 109  \t    \t\t                         \n            \n, 109492   \t,     10  \n\n   \t  \n                                         \t\t\t\t\t\t,      11\n  , 5,29 \n\t, \n269  \t\t\t \t\t\t \t \n        \n\n\n\n\n\n\n\n\n\n\n\n  \t,14     , 26,1  \n\t\t\t\t, 38 \t\t                        \t     \t\t,    958, 683 ,  194  \t\t,\n291\t, 485 \n  \n\n        \n\n\n\n\n, 281,  197, 7     \t, 8\t\t\t\t\t\t\t, 1  ,94,0  \t  \n     ,   2750755 \t \t   \t \t,   1       \n    \t, 19     \n , 160  , 133,192    \n\n\n\n\n\n\n\n\n\n     \t\t\t\t, 39,\n27\n   , 1 \t                 , 59, \n \n\t\t\n3   , 2   \t      \t\t,\n   697    \t \t \t   \n     \t \n\n\n\n\n\n\n\n\n\n\n         ,  735  \t           \t\t\n\n\n \n    \t ,3518      \t, 109  \n        \t\t  , 4041 , 280000     \n      \t   \t\t   \t   \t\t\t\t    ,   1] \t    \n\n    \t\n\n\t\t\t\t\t\t\t     \n\n  \t          \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n                                                                                                                                                             \t \n             \n\n\n          \t\t\t\t                      },\n        {\n                                                                      \n   "r"   \n            \t   :\n\n127 \n,  "c"\n  \t \n\n     \n\n\n \t  \t                  \n        \n        \n                                                                                          \n          \n        \t\t\t\t\t\t                  \t \t\n\n\n                                                \n           \n    \n    \n        \n\n\n\n\n       \n    \n    \n                                            \t\t          \n\n                                           \n                 \n                \n:\n\n\n\n      \n                        \n                    \n \t\t\t\t             \n\n\n\n\n\n   \n                 [ 18          \t \t , 2  \t,9    \t\t\t     \t\t\t\t\t                                                                                                     \n  \n  \n       \n     \t \n       ]\n\n\n\n   \t   \t        \t\t \t  \t           \n    }   \n , \n\n         \n      \n\n\n\n\n\n       \t     \t\t\t\t\t\n\t\t\t\t\t\n    {                                                                       "r" \t\t\t\t\t\t\t        \n    \n\n   \n         \t\t\t\t\t\t\t  \n    \t\t\t\t\t       \n\n      \n \n\n\n\n      \t    \t  \n\t\t\n  \t   \t  \n  \n             \t      \n\t\n     \t\t           \t \t \n       \t                   \n\n\n\n\n\n         \n     : 8        \t   \t\t\n\n\n\n\n                            \n     \n,     \t\t"c"                                                                 \n\n\n    \t               \n  \n\t        \t   \t \t   \t    \n     :\n\n   \t\t\t        \t   \t\t\n\t\t\n\t\t\n   \t       []    \t\t\t\t\t\t\t\t\t\t\t    \n  \t                                                                       \n         \n\n\n  \t  \t   \t                  \n  \t }\n    \t, \n  { \n  "r"  \n\n\n  \t   :\n\n\n\n  \n\n   0  \t   \t \t\t\n  ,    \n  \n\t\t\t\t             \t\t        \t       "c": \n  \n  \t\t\t\t\t\t        \t    \t       \n \n  \n\t  \n  \n\n    \n         \t\n\n [0                                                                       \n\n  \t      ,  10697  ,  697  \n         \n    \t\t  \n  \t,\n\n249   \n   ,  5      \n    \n    \t  \n\n   , 25\n\n\n\n\n\n\n\n\n\n\n\n\n\n      \n    \n  \n   , 207 \n    \n ', role='assistant', tool_calls=None, reasoning_content=None), logprobs=None, finish_reason='stop')] prompt_logprobs=None

mellea/backends/huggingface.py:583: Exception
---------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------
=== 15:42:24-INFO ======
passing in model options when generating with an adapter; some model options may be overwritten / ignored
------------------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------------------
INFO     fancy_logger:huggingface.py:475 passing in model options when generating with an adapter; some model options may be overwritten / ignored
-------------------------------------------------------------------------------------------------------- Captured stdout teardown ---------------------------------------------------------------------------------------------------------
=== 15:54:48-INFO ======
Cleaning up test_core backend GPU memory...
=== 15:54:48-INFO ======
  Cleared LRU cache
=== 15:54:48-INFO ======
  Removed accelerate dispatch hooks
---------------------------------------------------------------------------------------------------------- Captured log teardown ----------------------------------------------------------------------------------------------------------
INFO     fancy_logger:conftest.py:348 Cleaning up test_core backend GPU memory...
INFO     fancy_logger:conftest.py:371   Cleared LRU cache
INFO     fancy_logger:conftest.py:408   Removed accelerate dispatch hooks
______________________________________________________________________________________________________ test_hallucination_detection _______________________________________________________________________________________________________

backend = <mellea.backends.huggingface.LocalHFBackend object at 0x1473939e0>

    @pytest.mark.qualitative
    def test_hallucination_detection(backend):
        """Verify that the hallucination detection intrinsic functions properly."""
        context, assistant_response, docs = _read_input_json("hallucination_detection.json")
        expected = _read_output_json("hallucination_detection.json")
    
        # First call triggers adapter loading
        result = rag.flag_hallucinated_content(assistant_response, docs, context, backend)
        # pytest.approx() chokes on lists of records, so we do this complicated dance.
        for r, e in zip(result, expected, strict=True):  # type: ignore
>           assert pytest.approx(r, abs=2e-2) == e
E           AssertionError: assert approx({'resp...he sentence.}) == {'explanation...end': 31, ...}
E             
E             comparison failed. Mismatched elements: 1 / 5:
E             Max absolute difference: 0.022802131238099044
E             Max relative difference: 0.03036794087969006
E             Index                   | Obtained           | Expected                 
E             faithfulness_likelihood | 0.7280598165124975 | 0.7508619477505966 ± 0.02

test/stdlib/components/intrinsic/test_rag.py:159: AssertionError
---------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------
=== 15:56:26-INFO ======
passing in model options when generating with an adapter; some model options may be overwritten / ignored
------------------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------------------
INFO     fancy_logger:huggingface.py:475 passing in model options when generating with an adapter; some model options may be overwritten / ignored
============================================================================================================ warnings summary =============================================================================================================
test/backends/test_litellm_ollama.py::test_litellm_ollama_chat
test/backends/test_litellm_ollama.py::test_generate_from_raw
test/backends/test_litellm_ollama.py::test_async_parallel_requests
test/backends/test_litellm_ollama.py::test_async_avalue
test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[non-streaming]
test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[streaming]
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/aiohttp/connector.py:993: DeprecationWarning: enable_cleanup_closed ignored because https://github.com/python/cpython/pull/118960 is fixed in Python version sys.version_info(major=3, minor=12, micro=8, releaselevel='final', serial=0)
    super().__init__(

test/backends/test_litellm_ollama.py::test_litellm_ollama_chat
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='The answ...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_litellm_ollama.py::test_litellm_ollama_instruct
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content="Subject:...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_litellm_ollama.py::test_litellm_ollama_instruct
test/backends/test_litellm_ollama.py::test_litellm_ollama_instruct_options
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='yes', ro...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_litellm_ollama.py::test_litellm_ollama_instruct_options
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='Subject:...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_litellm_ollama.py::test_gen_slot
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='{\n  "re...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_litellm_ollama.py::test_async_parallel_requests
test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[streaming]
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/streaming_handler.py:1855: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
    obj_dict = processed_chunk.dict()

test/backends/test_litellm_ollama.py::test_async_parallel_requests
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='Goodbye!...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_litellm_ollama.py::test_async_parallel_requests
test/backends/test_litellm_ollama.py::test_async_avalue
test/backends/test_litellm_ollama.py::test_async_avalue
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='Hello! H...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/backends/test_tool_calls.py::test_tool_called_from_context_action
  <frozen abc>:106: DeprecationWarning: Use BaseMetaSerializer() instead.

test/backends/test_vision_ollama.py::test_image_block_construction
  /Users/ajbozarth/workspace/ai/mellea/test/backends/test_vision_ollama.py:38: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15)
    random_image = Image.fromarray(random_pixel_data, "RGB")

test/backends/test_vision_openai.py::test_image_block_construction
  /Users/ajbozarth/workspace/ai/mellea/test/backends/test_vision_openai.py:48: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15)
    random_image = Image.fromarray(random_pixel_data, "RGB")

test/cli/test_alora_train.py::test_alora_config_creation
test/cli/test_alora_train.py::test_lora_config_creation
test/cli/test_alora_train.py::test_invocation_prompt_tokenization
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/trl/trainer/sft_config.py:257: DeprecationWarning: `max_seq_length` is deprecated and will be removed in version 0.20.0. Use `max_length` instead.
    warnings.warn(

test/helpers/test_event_loop_helper.py::test_event_loop_handler_with_forking
  /Users/ajbozarth/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=18760) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

test/stdlib/components/docs/test_richdocument.py::test_richdocument_basics
test/stdlib/components/docs/test_richdocument.py::test_richdocument_markdown
test/stdlib/components/docs/test_richdocument.py::test_richdocument_save
test/stdlib/components/docs/test_richdocument.py::test_richdocument_save
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/docling_core/transforms/serializer/markdown.py:490: DeprecationWarning: Field `annotations` is deprecated; use `meta` instead.
    for ann in item.annotations

test/stdlib/components/intrinsic/test_core.py: 2 warnings
test/stdlib/components/intrinsic/test_guardian.py: 3 warnings
test/stdlib/components/intrinsic/test_rag.py: 5 warnings
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/peft/tuners/tuners_utils.py:285: UserWarning: Already found a `peft_config` attribute in the model. This will lead to having multiple adapters in the model. Make sure to know what you are doing!
    warnings.warn(

test/stdlib/test_spans.py::test_lazy_spans
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/torch/nn/functional.py:5294: UserWarning: MPS: The constant padding of more than 3 dimensions is not currently supported natively. It uses View Ops default implementation to run. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/operations/Pad.mm:468.)
    return torch._C._nn.pad(input, pad, mode, value)

test/telemetry/test_logging.py::test_otlp_logging_enabled_without_endpoint_warns
  /Users/ajbozarth/workspace/ai/mellea/mellea/telemetry/logging.py:97: UserWarning: OTLP logs exporter is enabled (MELLEA_LOGS_OTLP=true) but no endpoint is configured. Set OTEL_EXPORTER_OTLP_LOGS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT to export logs.
    _logger_provider = _setup_logger_provider()

test/telemetry/test_metrics.py: 24 warnings
test/telemetry/test_metrics_backend.py: 8 warnings
test/telemetry/test_metrics_latency.py: 3 warnings
test/telemetry/test_metrics_token.py: 4 warnings
  /Users/ajbozarth/workspace/ai/mellea/mellea/telemetry/metrics.py:266: UserWarning: Metrics are enabled (MELLEA_METRICS_ENABLED=true) but no exporters are configured. Metrics will be collected but not exported. Set MELLEA_METRICS_PROMETHEUS=true, set MELLEA_METRICS_OTLP=true with an endpoint (OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT), or set MELLEA_METRICS_CONSOLE=true to export metrics.
    _meter_provider = _setup_meter_provider()

test/telemetry/test_metrics.py: 28 warnings
test/telemetry/test_metrics_backend.py: 8 warnings
test/telemetry/test_metrics_latency.py: 3 warnings
test/telemetry/test_metrics_token.py: 4 warnings
  /Users/ajbozarth/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/importlib/__init__.py:131: UserWarning: TokenMetricsPlugin already registered: Plugin token_metrics.generation_post_call already registered
    _bootstrap._exec(spec, module)

test/telemetry/test_metrics.py: 28 warnings
test/telemetry/test_metrics_backend.py: 8 warnings
test/telemetry/test_metrics_latency.py: 3 warnings
test/telemetry/test_metrics_token.py: 4 warnings
  /Users/ajbozarth/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/importlib/__init__.py:131: UserWarning: LatencyMetricsPlugin already registered: Plugin latency_metrics.generation_post_call already registered
    _bootstrap._exec(spec, module)

test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[non-streaming]
test/telemetry/test_tracing.py::test_session_with_tracing_disabled
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content='Understo...ields={'refusal': None}), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...ider_specific_fields={}), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[streaming]
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/model_response_utils.py:206: PydanticDeprecatedSince211: Accessing the 'model_computed_fields' attribute on the instance is deprecated. Instead, you should access this attribute from the model class. Deprecated in Pydantic V2.11 to be removed in V3.0.
    or callable(getattr(delta, attr_name))

test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[streaming]
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/litellm/litellm_core_utils/model_response_utils.py:206: PydanticDeprecatedSince211: Accessing the 'model_fields' attribute on the instance is deprecated. Instead, you should access this attribute from the model class. Deprecated in Pydantic V2.11 to be removed in V3.0.
    or callable(getattr(delta, attr_name))

test/telemetry/test_metrics_backend.py::test_litellm_token_metrics_integration[streaming]
  /Users/ajbozarth/workspace/ai/mellea/.venv/lib/python3.12/site-packages/pydantic/main.py:464: UserWarning: Pydantic serializer warnings:
    PydanticSerializationUnexpectedValue(Expected 10 fields but got 5: Expected `Message` - serialized value may not be as expected [field_name='message', input_value=Message(content="In terms...er_specific_fields=None), input_type=Message])
    PydanticSerializationUnexpectedValue(Expected `StreamingChoices` - serialized value may not be as expected [field_name='choices', input_value=Choices(finish_reason='st...r_specific_fields=None)), input_type=Choices])
    return self.__pydantic_serializer__.to_python(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================================ Skipped Examples =============================================================================================================
The following examples were skipped during collection:

  • 102_example.py: Example marked with skip marker
  • example_readme_generator.py: Example marked with skip marker
  • make_training_data.py: Example marked with skip marker
  • stembolts_intrinsic.py: Example marked with skip marker
  • bedrock_litellm_example.py: Example marked with skip marker
  • bedrock_openai_example.py: Example marked with skip marker
  • qiskit_code_validation.py: Example marked with skip marker
  • validation_helpers.py: Example marked with skip marker
  • python_decompose_result.py: Example marked to always skip (skip_always marker)
  • m_decomp_result.py: Example marked to always skip (skip_always marker)
  • client.py: Example marked to always skip (skip_always marker)
  • pii_serve.py: Example marked to always skip (skip_always marker)
  • mcp_example.py: Example marked to always skip (skip_always marker)
  • rich_document_advanced.py: Example marked with skip marker
  • mellea_pdf.py: Example marked to always skip (skip_always marker)
  • simple_rag_with_filter.py: Example marked to always skip (skip_always marker)
============================================================================================================= tests coverage ==============================================================================================================
____________________________________________________________________________________________ coverage: platform darwin, python 3.12.8-final-0 _____________________________________________________________________________________________

Coverage HTML written to dir htmlcov
Coverage JSON written to file coverage.json
========================================================================================================= short test summary info =========================================================================================================
FAILED test/backends/test_litellm_ollama.py::test_async_avalue - assert 503.11499999999995 is None
FAILED test/backends/test_ollama.py::test_async_avalue - assert 260.718 is None
FAILED test/backends/test_openai_ollama.py::test_async_avalue - assert 336.167 is None
FAILED test/stdlib/components/intrinsic/test_core.py::test_find_context_attributions - Exception: Intrinsic did not return a JSON: choices=[ChatCompletionResponseChoice(index=0, message=AssistantMessage(content='[ \t\t\t\t\t\t\t\t\t\t\t\t              \n                           \n    \t             \t\t\t\t\t \n  ...
FAILED test/stdlib/components/intrinsic/test_rag.py::test_hallucination_detection - AssertionError: assert approx({'resp...he sentence.}) == {'explanation...end': 31, ...}
================================================================ 5 failed, 804 passed, 61 skipped, 19 deselected, 2 xfailed, 1 xpassed, 171 warnings in 1699.07s (0:28:19) ================================================================

[planetf1]

This may help keep memory usage more in control - though does not directly address this issue with how space is calculated
Will revisit the log above once happy with the eviction in #804