runtime: add session-level observability plumbing#1436
runtime: add session-level observability plumbing#1436Pnkcaht wants to merge 5 commits intodocker:mainfrom
Conversation
Signed-off-by: pnkcaht <samzoovsk19@gmail.com>
|
ping @dgageot : I am not sure but I think this one could help with making sure we do not regress gordon v2 system prompt caching |
|
I'm not sure how useful this is, we have OTEL already for example, why not use that? |
Signed-off-by: pnkcaht <samzoovsk19@gmail.com>
|
Hi everyone :) @rumpl @simonferquel Sorry for the delay, I was working on other pull requests. This work is motivated by and intentionally focuses on session-level invariants, rather than general observability. If there's a better mechanism for validating these types of session-level invariants, we'd gladly adopt it. However, the goal here is specifically to fill the gap described in #1345, not to introduce or extend an observability stack. This PR is also a first step in that direction; additional metrics and configurations can be added later. New changesThis pull request had encountered an error in the Check, specifically related to the name SessionMetrics in the session.go and session_manager.go files. Basically, I changed the name SessionMetrics to just Metrics, which has already passed the test. |
Signed-off-by: pnkcaht <samzoovsk19@gmail.com>
What I did
Introduced session-level observability plumbing to collect and emit high-level interaction and performance metrics during agent execution.
This change adds support for tracking session metrics such as user messages, assistant messages, tool calls, tool errors, execution start/end times, and token usage totals. These metrics are emitted as ephemeral runtime events and are not persisted nor used for billing, enabling better observability without impacting existing session storage semantics.
The implementation wires metrics collection through the session manager, runtime events, and protocol layer, making the data available to consumers such as the TUI and future UI integrations.
Related issue
Related to the feature request for observability and interaction summaries:
Refs #1345
What’s missing / follow-up work
This PR intentionally focuses on infrastructure and data flow. The following items are out of scope and left for follow-up changes:
These can now be implemented incrementally on top of the metrics exposed here.
Screenshot (Explain)
Observability Architecture Overview
The provided diagram illustrates the new session-level observability plumbing. Key highlights of the flow include:
Lifecycle Tracking: Upon triggering execution, the Session Manager initializes session metrics (marking StartedAt) and manages the execution lifecycle.
Real-time Collection: A dedicated Goroutine monitors the event stream from the LLM Provider. It intercepts assistant messages, tool calls, and tool errors to increment performance counters in real-time.
Ephemeral Events: These metrics are emitted as structured Runtime Events via the Protocol Layer (Protobuf) for consumption by the TUI/UI.
Zero Persistence: While the core session state is saved to the Bank (database), the metrics are explicitly excluded from JSON serialization (json:"-"), ensuring they remain ephemeral and do not impact storage or billing.
Final Handshake: At the end of the execution, EndedAt is recorded, and a final TokenUsageEvent is emitted to provide a complete summary of the interaction.
Before
After