Streaming should return new chunks

[nrfulton]

Currently streaming returns the entire generation so far:

from mellea import start_session
from mellea.stdlib.base import CBlock
from mellea.backends.types import ModelOption

import asyncio


async def stream_chat(prompt: str) -> str:
    m = start_session()

    mot, _ = await m.backend.generate_from_context(
        action=CBlock(value=prompt), ctx=m.ctx, model_options={ModelOption.STREAM: True}
    )

    while not mot.is_computed():
        print(await mot.astream(), flush=True)

    print("\n\nFINAL ANSWER")
    print(mot.value)

    return str(mot.value)


if __name__ == "__main__":
    asyncio.run(stream_chat("Write a tight 8-line poem about granite and winter."))

Instead, we should be returning only the new text.

We could also return a StreamingResult object that contains both. But getting just the latest diff should easy.

[akihikokuroda]

May I take this?

[akihikokuroda]

Output is like this:

Starting Mellea session: backend=ollama, model=granite4:micro, context=SimpleContext
In stone's embrace
 where frost does dwell
,
Beneath
 the snow, in
 heart of cliff.

Winter's chill on
 granite told,
Secret
s only ice could
 fill.

Stone stands
 firm while snow may
 shiver,
Anc
ient tales within it
 stiver,
A
 testament to time,
 a whisperer,

Of seasons' dance
 and earth's deep
In stone's embrace where frost does dwell,
Beneath the snow, in heart of cliff.
Winter's chill on granite told,
Secrets only ice could fill.

Stone stands firm while snow may shiver,
Ancient tales within it stiver,
A testament to time, a whisperer,
Of seasons' dance and earth's deep treasure.


FINAL ANSWER
In stone's embrace where frost does dwell,
Beneath the snow, in heart of cliff.
Winter's chill on granite told,
Secrets only ice could fill.

Stone stands firm while snow may shiver,
Ancient tales within it stiver,
A testament to time, a whisperer,
Of seasons' dance and earth's deep treasure.