From 878c357acda8d4e2be358ab299d2238dc7352293 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 10:59:19 -0800 Subject: [PATCH 01/19] Add OpenTelemetry observability support - Add instrumentation module with span building for chat completions and tool calls - Support session tracking across conversation turns - Add configurable metadata prefix for custom trace attributes - Implement NullTracer pattern for graceful fallback when OTel not installed - Fail loudly with helpful error when tracing enabled but OTel unavailable - Add comprehensive documentation for LangSmith and other backends - Include full test coverage for instrumentation --- Appraisals | 8 +- Gemfile | 7 +- docs/_advanced/observability.md | 274 +++++++++ gemfiles/rails_7.1.gemfile | 5 +- gemfiles/rails_7.2.gemfile | 5 +- gemfiles/rails_8.0.gemfile | 5 +- gemfiles/rails_8.1.gemfile | 5 +- lib/ruby_llm/active_record/acts_as_legacy.rb | 5 +- lib/ruby_llm/active_record/chat_methods.rb | 3 +- lib/ruby_llm/chat.rb | 134 ++++- lib/ruby_llm/configuration.rb | 12 +- lib/ruby_llm/instrumentation.rb | 209 +++++++ lib/ruby_llm/tool.rb | 2 +- ...instrumentation_creates_spans_for_chat.yml | 116 ++++ .../instrumentation_creates_tool_spans.yml | 213 +++++++ .../instrumentation_includes_token_usage.yml | 116 ++++ .../instrumentation_maintains_session_id.yml | 198 +++++++ .../active_record/acts_as_model_spec.rb | 6 +- spec/ruby_llm/instrumentation_spec.rb | 541 ++++++++++++++++++ 19 files changed, 1839 insertions(+), 25 deletions(-) create mode 100644 docs/_advanced/observability.md create mode 100644 lib/ruby_llm/instrumentation.rb create mode 100644 spec/fixtures/vcr_cassettes/instrumentation_creates_spans_for_chat.yml create mode 100644 spec/fixtures/vcr_cassettes/instrumentation_creates_tool_spans.yml create mode 100644 spec/fixtures/vcr_cassettes/instrumentation_includes_token_usage.yml create mode 100644 spec/fixtures/vcr_cassettes/instrumentation_maintains_session_id.yml create mode 100644 spec/ruby_llm/instrumentation_spec.rb diff --git a/Appraisals b/Appraisals index f953e00be..de459c5b2 100644 --- a/Appraisals +++ b/Appraisals @@ -1,25 +1,25 @@ # frozen_string_literal: true appraise 'rails-7.1' do - group :development do + group :development, :test do gem 'rails', '~> 7.1.0' end end appraise 'rails-7.2' do - group :development do + group :development, :test do gem 'rails', '~> 7.2.0' end end appraise 'rails-8.0' do - group :development do + group :development, :test do gem 'rails', '~> 8.0.0' end end appraise 'rails-8.1' do - group :development do + group :development, :test do gem 'rails', '~> 8.1.0' end end diff --git a/Gemfile b/Gemfile index b9a7dcc19..c95952ce5 100644 --- a/Gemfile +++ b/Gemfile @@ -4,14 +4,14 @@ source 'https://rubygems.org' gemspec -group :development do # rubocop:disable Metrics/BlockLength +group :development, :test do # rubocop:disable Metrics/BlockLength gem 'appraisal' gem 'async', platform: :mri gem 'bundler', '>= 2.0' gem 'codecov' gem 'dotenv' gem 'ferrum' - gem 'flay' + gem 'flay', '< 2.14' # 2.14 switched from ruby_parser to prism, causing CI issues gem 'image_processing', '~> 1.2' gem 'irb' gem 'json-schema' @@ -39,4 +39,7 @@ group :development do # rubocop:disable Metrics/BlockLength # Optional dependency for Vertex AI gem 'googleauth' + + # OpenTelemetry for observability testing + gem 'opentelemetry-sdk' end diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md new file mode 100644 index 000000000..6c3b2726b --- /dev/null +++ b/docs/_advanced/observability.md @@ -0,0 +1,274 @@ +--- +layout: default +title: Observability +nav_order: 7 +description: Send traces to LangSmith, DataDog, or any OpenTelemetry backend. Monitor your LLM usage in production. +redirect_from: + - /guides/observability +--- + +# {{ page.title }} +{: .no_toc } + +{{ page.description }} +{: .fs-6 .fw-300 } + +## Table of contents +{: .no_toc .text-delta } + +1. TOC +{:toc} + +--- + +After reading this guide, you will know: + +* How to enable OpenTelemetry tracing in RubyLLM +* How to configure backends like LangSmith, DataDog, and Jaeger +* How session tracking groups multi-turn conversations +* How to add custom metadata to traces +* What attributes are captured in spans + +## What's Supported + +| Feature | Status | +|---------|--------| +| Chat completions | ✅ Supported | +| Tool calls | ✅ Supported | +| Session tracking | ✅ Supported | +| Content logging (opt-in) | ✅ Supported | +| Streaming | ❌ Not yet supported | +| Embeddings | ❌ Not yet supported | +| Image generation | ❌ Not yet supported | +| Transcription | ❌ Not yet supported | +| Moderation | ❌ Not yet supported | + +--- + +## Quick Start + +### 1. Install OpenTelemetry gems + +```ruby +# Gemfile +gem 'opentelemetry-sdk' +gem 'opentelemetry-exporter-otlp' +``` + +### 2. Enable tracing in RubyLLM + +```ruby +RubyLLM.configure do |config| + config.tracing_enabled = true +end +``` + +### 3. Configure your exporter + +See [Backend Setup](#backend-setup) for LangSmith, DataDog, Jaeger, etc. + +--- + +## Configuration Options + +```ruby +RubyLLM.configure do |config| + # Enable tracing (default: false) + config.tracing_enabled = true + + # Log prompt/completion content (default: false) + # Enable for full LangSmith functionality + config.tracing_log_content = true + + # Max content length before truncation (default: 10000) + config.tracing_max_content_length = 5000 +end +``` + +> **Privacy note:** `tracing_log_content` sends your prompts and completions to your tracing backend. Only enable this if you're comfortable with your backend seeing this data. +{: .warning } + +### Service Name + +Your service name identifies your application in the tracing backend. Set it via environment variable: + +```bash +export OTEL_SERVICE_NAME="my_app" +``` + +### Custom Metadata + +You can attach custom metadata to traces for filtering and debugging: + +```ruby +chat = RubyLLM.chat + .with_metadata(user_id: current_user.id, request_id: request.uuid) +chat.ask("Hello!") +``` + +Metadata appears as `metadata.*` attributes by default. For LangSmith's metadata panel, set: + +```ruby +RubyLLM.configure do |config| + config.tracing_metadata_prefix = 'langsmith.metadata' +end +``` + +--- + +## Backend Setup + +### LangSmith + +LangSmith is LangChain's observability platform with specialized LLM debugging features. + +```ruby +# config/initializers/ruby_llm.rb +RubyLLM.configure do |config| + config.tracing_enabled = true + config.tracing_log_content = true + config.tracing_metadata_prefix = 'langsmith.metadata' +end +``` + +```ruby +# config/initializers/opentelemetry.rb +require 'opentelemetry/sdk' +require 'opentelemetry/exporter/otlp' + +OpenTelemetry::SDK.configure do |c| + c.add_span_processor( + OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new( + OpenTelemetry::Exporter::OTLP::Exporter.new( + endpoint: 'https://api.smith.langchain.com/otel/v1/traces', + headers: { + 'x-api-key' => 'lsv2_pt_...', + 'Langsmith-Project' => 'my-project' + } + ) + ) + ) +end +``` + +LangSmith uses the `Langsmith-Project` header (not `service_name`) to organize traces. + +### Other Backends + +RubyLLM works with any OpenTelemetry-compatible backend. Configure the `opentelemetry-exporter-otlp` gem to send traces to your platform of choice. + +> Using DataDog, Jaeger, Honeycomb, or another platform? Consider [contributing](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md) a setup example! +{: .note } + +--- + +## What Gets Traced + +### Chat Completions + +Each call to `chat.ask()` creates a `ruby_llm.chat` span with: + +| Attribute | Description | +|-----------|-------------| +| `gen_ai.system` | Provider name (openai, anthropic, etc.) | +| `gen_ai.request.model` | Requested model ID | +| `gen_ai.request.temperature` | Temperature setting (if specified) | +| `gen_ai.response.model` | Actual model used | +| `gen_ai.usage.input_tokens` | Input token count | +| `gen_ai.usage.output_tokens` | Output token count | +| `gen_ai.conversation.id` | Session ID for grouping conversations | + +### Tool Calls + +When tools are invoked, child `ruby_llm.tool` spans are created with: + +| Attribute | Description | +|-----------|-------------| +| `gen_ai.tool.name` | Name of the tool | +| `gen_ai.tool.call.id` | Unique call identifier | +| `input.value` | Tool arguments (if content logging enabled) | +| `output.value` | Tool result (if content logging enabled) | + +### Content Logging + +When `tracing_log_content = true`, prompts and completions are logged: + +| Attribute | Description | +|-----------|-------------| +| `gen_ai.prompt.0.role` | Role of first message (user, system, assistant) | +| `gen_ai.prompt.0.content` | Content of first message | +| `gen_ai.completion.0.role` | Role of response | +| `gen_ai.completion.0.content` | Response content | + +--- + +## Session Tracking + +Each `Chat` instance gets a unique `session_id`. All traces from that chat include this ID: + +```ruby +chat = RubyLLM.chat +chat.ask("Hello") # session_id: f47ac10b-58cc-4372-a567-0e02b2c3d479 +chat.ask("How are you?") # session_id: f47ac10b-58cc-4372-a567-0e02b2c3d479 (same) + +chat2 = RubyLLM.chat +chat2.ask("Hi") # session_id: 7c9e6679-7425-40de-944b-e07fc1f90ae7 (different) +``` + +### Custom Session IDs + +For applications that persist conversations, pass your own session ID to group related traces: + +```ruby +chat = RubyLLM.chat(session_id: conversation.id) +chat.ask("Hello") + +# Later, when user continues the conversation: +chat = RubyLLM.chat(session_id: conversation.id) +chat.ask("Follow up") # Same session_id, grouped together +``` + +--- + +## Troubleshooting + +### "I don't see any traces" + +1. Verify `config.tracing_enabled = true` is set +2. Check your OpenTelemetry exporter configuration +3. Ensure the `opentelemetry-sdk` gem is installed +4. Check your backend's API key and endpoint + +### "I see traces but no content" + +Enable content logging: + +```ruby +RubyLLM.configure do |config| + config.tracing_log_content = true +end +``` + +### "My tracing backend is getting too much data" + +1. Reduce `tracing_max_content_length` to truncate large messages +2. Disable content logging: `config.tracing_log_content = false` +3. Configure sampling via environment variables: + +```bash +# Sample only 10% of traces +export OTEL_TRACES_SAMPLER="traceidratio" +export OTEL_TRACES_SAMPLER_ARG="0.1" +``` + +### "Traces aren't grouped in LangSmith" + +Make sure you're reusing the same `Chat` instance for multi-turn conversations. Each `Chat.new` creates a new session. + +## Next Steps + +* [Chatting with AI Models]({% link _core_features/chat.md %}) +* [Using Tools]({% link _core_features/tools.md %}) +* [Rails Integration]({% link _advanced/rails.md %}) +* [Error Handling]({% link _advanced/error-handling.md %}) + diff --git a/gemfiles/rails_7.1.gemfile b/gemfiles/rails_7.1.gemfile index 10ea3193d..5ebf2436c 100644 --- a/gemfiles/rails_7.1.gemfile +++ b/gemfiles/rails_7.1.gemfile @@ -2,14 +2,14 @@ source "https://rubygems.org" -group :development do +group :development, :test do gem "appraisal" gem "async", platform: :mri gem "bundler", ">= 2.0" gem "codecov" gem "dotenv" gem "ferrum" - gem "flay" + gem "flay", "< 2.14" gem "image_processing", "~> 1.2" gem "irb" gem "json-schema" @@ -32,6 +32,7 @@ group :development do gem "vcr" gem "webmock", "~> 3.18" gem "googleauth" + gem "opentelemetry-sdk" end gemspec path: "../" diff --git a/gemfiles/rails_7.2.gemfile b/gemfiles/rails_7.2.gemfile index f0f87e803..c4b3915ee 100644 --- a/gemfiles/rails_7.2.gemfile +++ b/gemfiles/rails_7.2.gemfile @@ -2,14 +2,14 @@ source "https://rubygems.org" -group :development do +group :development, :test do gem "appraisal" gem "async", platform: :mri gem "bundler", ">= 2.0" gem "codecov" gem "dotenv" gem "ferrum" - gem "flay" + gem "flay", "< 2.14" gem "image_processing", "~> 1.2" gem "irb" gem "json-schema" @@ -32,6 +32,7 @@ group :development do gem "vcr" gem "webmock", "~> 3.18" gem "googleauth" + gem "opentelemetry-sdk" end gemspec path: "../" diff --git a/gemfiles/rails_8.0.gemfile b/gemfiles/rails_8.0.gemfile index 80e2c2c51..a8286fefb 100644 --- a/gemfiles/rails_8.0.gemfile +++ b/gemfiles/rails_8.0.gemfile @@ -2,14 +2,14 @@ source "https://rubygems.org" -group :development do +group :development, :test do gem "appraisal" gem "async", platform: :mri gem "bundler", ">= 2.0" gem "codecov" gem "dotenv" gem "ferrum" - gem "flay" + gem "flay", "< 2.14" gem "image_processing", "~> 1.2" gem "irb" gem "json-schema" @@ -32,6 +32,7 @@ group :development do gem "vcr" gem "webmock", "~> 3.18" gem "googleauth" + gem "opentelemetry-sdk" end gemspec path: "../" diff --git a/gemfiles/rails_8.1.gemfile b/gemfiles/rails_8.1.gemfile index b7dc8724c..257f736c9 100644 --- a/gemfiles/rails_8.1.gemfile +++ b/gemfiles/rails_8.1.gemfile @@ -2,14 +2,14 @@ source "https://rubygems.org" -group :development do +group :development, :test do gem "appraisal" gem "async", platform: :mri gem "bundler", ">= 2.0" gem "codecov" gem "dotenv" gem "ferrum" - gem "flay" + gem "flay", "< 2.14" gem "image_processing", "~> 1.2" gem "irb" gem "json-schema" @@ -32,6 +32,7 @@ group :development do gem "vcr" gem "webmock", "~> 3.18" gem "googleauth" + gem "opentelemetry-sdk" end gemspec path: "../" diff --git a/lib/ruby_llm/active_record/acts_as_legacy.rb b/lib/ruby_llm/active_record/acts_as_legacy.rb index 97679c126..abc43880d 100644 --- a/lib/ruby_llm/active_record/acts_as_legacy.rb +++ b/lib/ruby_llm/active_record/acts_as_legacy.rb @@ -88,10 +88,11 @@ module ChatLegacyMethods def to_llm(context: nil) # model_id is a string that RubyLLM can resolve + # session_id uses the AR record ID for tracing session grouping @chat ||= if context - context.chat(model: model_id) + context.chat(model: model_id, session_id: id) else - RubyLLM.chat(model: model_id) + RubyLLM.chat(model: model_id, session_id: id) end @chat.reset_messages! diff --git a/lib/ruby_llm/active_record/chat_methods.rb b/lib/ruby_llm/active_record/chat_methods.rb index 41930548c..f95269eb5 100644 --- a/lib/ruby_llm/active_record/chat_methods.rb +++ b/lib/ruby_llm/active_record/chat_methods.rb @@ -79,7 +79,8 @@ def to_llm model_record = model_association @chat ||= (context || RubyLLM).chat( model: model_record.model_id, - provider: model_record.provider.to_sym + provider: model_record.provider.to_sym, + session_id: id # Use AR record ID for tracing session grouping ) @chat.reset_messages! diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index d03d872ca..e6c46cb40 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -5,9 +5,9 @@ module RubyLLM class Chat include Enumerable - attr_reader :model, :messages, :tools, :params, :headers, :schema + attr_reader :model, :messages, :tools, :params, :headers, :schema, :session_id, :metadata - def initialize(model: nil, provider: nil, assume_model_exists: false, context: nil) + def initialize(model: nil, provider: nil, assume_model_exists: false, context: nil, session_id: nil) if assume_model_exists && !provider raise ArgumentError, 'Provider must be specified if assume_model_exists is true' end @@ -16,6 +16,8 @@ def initialize(model: nil, provider: nil, assume_model_exists: false, context: n @config = context&.config || RubyLLM.config model_id = model || @config.default_model with_model(model_id, provider: provider, assume_exists: assume_model_exists) + @session_id = session_id || SecureRandom.uuid + @metadata = {} @temperature = nil @messages = [] @tools = {} @@ -84,6 +86,11 @@ def with_headers(**headers) self end + def with_metadata(**metadata) + @metadata = @metadata.merge(metadata) + self + end + def with_schema(schema) schema_instance = schema.is_a?(Class) ? schema.new : schema @@ -121,7 +128,18 @@ def each(&) messages.each(&) end - def complete(&) # rubocop:disable Metrics/PerceivedComplexity + def complete(&) + # Skip instrumentation for streaming (not supported yet) + return complete_without_instrumentation(&) if block_given? + + Instrumentation.tracer.in_span('ruby_llm.chat', kind: Instrumentation::SpanKind::CLIENT) do |span| + complete_with_span(span, &) + end + end + + private + + def complete_without_instrumentation(&) response = @provider.complete( messages, tools: @tools, @@ -133,6 +151,73 @@ def complete(&) # rubocop:disable Metrics/PerceivedComplexity &wrap_streaming_block(&) ) + finalize_response(response, &) + end + + def complete_with_span(span, &) + # Set request attributes + if span.recording? + span.add_attributes( + Instrumentation::SpanBuilder.build_request_attributes( + model: @model, + provider: @provider.slug, + session_id: @session_id, + temperature: @temperature, + metadata: @metadata + ) + ) + + # Log message content if enabled + if @config.tracing_log_content + span.add_attributes( + Instrumentation::SpanBuilder.build_message_attributes( + messages, + max_length: @config.tracing_max_content_length + ) + ) + end + end + + response = @provider.complete( + messages, + tools: @tools, + temperature: @temperature, + model: @model, + params: @params, + headers: @headers, + schema: @schema + ) + + # Add response attributes + if span.recording? + span.add_attributes(Instrumentation::SpanBuilder.build_response_attributes(response)) + + if @config.tracing_log_content + span.add_attributes( + Instrumentation::SpanBuilder.build_completion_attributes( + response, + max_length: @config.tracing_max_content_length + ) + ) + end + end + + finalize_response(response, &) + rescue StandardError => e + record_span_error(span, e) + raise + end + + def record_span_error(span, exception) + return unless span.recording? + + span.record_exception(exception) + return unless defined?(OpenTelemetry::Trace::Status) + + span.status = OpenTelemetry::Trace::Status.error(exception.message) + end + + def finalize_response(response, &) # rubocop:disable Metrics/PerceivedComplexity @on[:new_message]&.call unless block_given? if @schema && response.content.is_a?(String) @@ -153,6 +238,8 @@ def complete(&) # rubocop:disable Metrics/PerceivedComplexity end end + public + def add_message(message_or_attributes) message = message_or_attributes.is_a?(Message) ? message_or_attributes : Message.new(message_or_attributes) messages << message @@ -205,9 +292,48 @@ def handle_tool_calls(response, &) # rubocop:disable Metrics/PerceivedComplexity end def execute_tool(tool_call) + Instrumentation.tracer.in_span('ruby_llm.tool', kind: Instrumentation::SpanKind::INTERNAL) do |span| + execute_tool_with_span(tool_call, span) + end + end + + def execute_tool_with_span(tool_call, span) tool = tools[tool_call.name.to_sym] args = tool_call.arguments - tool.call(args) + + if span.recording? + span.add_attributes( + Instrumentation::SpanBuilder.build_tool_attributes( + tool_call: tool_call, + session_id: @session_id + ) + ) + + if @config.tracing_log_content + span.add_attributes( + Instrumentation::SpanBuilder.build_tool_input_attributes( + tool_call: tool_call, + max_length: @config.tracing_max_content_length + ) + ) + end + end + + result = tool.call(args) + + if span.recording? && @config.tracing_log_content + span.add_attributes( + Instrumentation::SpanBuilder.build_tool_output_attributes( + result: result, + max_length: @config.tracing_max_content_length + ) + ) + end + + result + rescue StandardError => e + record_span_error(span, e) + raise end def build_content(message, attachments) diff --git a/lib/ruby_llm/configuration.rb b/lib/ruby_llm/configuration.rb index e1c12902a..4f7218a54 100644 --- a/lib/ruby_llm/configuration.rb +++ b/lib/ruby_llm/configuration.rb @@ -46,7 +46,12 @@ class Configuration :logger, :log_file, :log_level, - :log_stream_debug + :log_stream_debug, + # Tracing configuration + :tracing_enabled, + :tracing_log_content, + :tracing_max_content_length, + :tracing_metadata_prefix def initialize @request_timeout = 300 @@ -69,6 +74,11 @@ def initialize @log_file = $stdout @log_level = ENV['RUBYLLM_DEBUG'] ? Logger::DEBUG : Logger::INFO @log_stream_debug = ENV['RUBYLLM_STREAM_DEBUG'] == 'true' + + @tracing_enabled = false + @tracing_log_content = false + @tracing_max_content_length = 10_000 + @tracing_metadata_prefix = 'metadata' end def instance_variables diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb new file mode 100644 index 000000000..37729a4dd --- /dev/null +++ b/lib/ruby_llm/instrumentation.rb @@ -0,0 +1,209 @@ +# frozen_string_literal: true + +require 'singleton' + +module RubyLLM + # OpenTelemetry instrumentation for RubyLLM + # Provides tracing capabilities when enabled and OpenTelemetry is available + module Instrumentation + # Span kind constants (matches OpenTelemetry::Trace::SpanKind) + module SpanKind + CLIENT = :client + INTERNAL = :internal + end + class << self + def enabled? + return false unless RubyLLM.config.tracing_enabled + + unless otel_available? + raise RubyLLM::ConfigurationError, <<~MSG.strip + Tracing is enabled but OpenTelemetry is not available. + Please add the following gems to your Gemfile: + + gem 'opentelemetry-sdk' + gem 'opentelemetry-exporter-otlp' + + Then run `bundle install` and configure OpenTelemetry in an initializer. + See https://rubyllm.com/advanced/observability for setup instructions. + MSG + end + + true + end + + def tracer + return NullTracer.instance unless enabled? + + @tracer ||= OpenTelemetry.tracer_provider.tracer( + 'ruby_llm', + RubyLLM::VERSION + ) + end + + def reset! + @tracer = nil + end + + private + + def otel_available? + return false unless defined?(OpenTelemetry) + + !!OpenTelemetry.tracer_provider + end + end + + # No-op tracer used when tracing is disabled or OpenTelemetry is not available + class NullTracer + include Singleton + + def in_span(_name, **_options) + yield NullSpan.instance + end + end + + # No-op span that responds to all span methods but does nothing + class NullSpan + include Singleton + + def recording? + false + end + + def set_attribute(_key, _value) + self + end + + def add_attributes(_attributes) + self + end + + def record_exception(_exception, _attributes = {}) + self + end + + def status=(_status) + # no-op + end + + def finish + self + end + end + + # Helper for building span attributes + module SpanBuilder + class << self + def truncate_content(content, max_length) + return nil if content.nil? + + content_str = content.to_s + return content_str if content_str.length <= max_length + + "#{content_str[0, max_length]}... [truncated]" + end + + def extract_content_text(content) + case content + when String + content + when RubyLLM::Content + content.text || describe_attachments(content.attachments) + else + content.to_s + end + end + + def describe_attachments(attachments) + return '[no content]' if attachments.empty? + + descriptions = attachments.map { |a| "#{a.type}: #{a.filename}" } + "[#{descriptions.join(', ')}]" + end + + def build_message_attributes(messages, max_length:) + attrs = {} + messages.each_with_index do |msg, idx| + attrs["gen_ai.prompt.#{idx}.role"] = msg.role.to_s + content = extract_content_text(msg.content) + attrs["gen_ai.prompt.#{idx}.content"] = truncate_content(content, max_length) + end + # Set input.value for LangSmith Input panel (last user message) + last_user_msg = messages.reverse.find { |m| m.role.to_s == 'user' } + if last_user_msg + content = extract_content_text(last_user_msg.content) + attrs['input.value'] = truncate_content(content, max_length) + end + attrs + end + + def build_completion_attributes(message, max_length:) + attrs = {} + attrs['gen_ai.completion.0.role'] = message.role.to_s + content = extract_content_text(message.content) + truncated = truncate_content(content, max_length) + attrs['gen_ai.completion.0.content'] = truncated + # Set output.value for LangSmith Output panel + attrs['output.value'] = truncated + attrs + end + + def build_request_attributes(model:, provider:, session_id:, temperature: nil, metadata: nil) + attrs = { + 'langsmith.span.kind' => 'LLM', + 'gen_ai.system' => provider.to_s, + 'gen_ai.operation.name' => 'chat', + 'gen_ai.request.model' => model.id, + 'gen_ai.conversation.id' => session_id + } + attrs['gen_ai.request.temperature'] = temperature if temperature + build_metadata_attributes(attrs, metadata) if metadata + attrs + end + + def build_metadata_attributes(attrs, metadata, prefix: RubyLLM.config.tracing_metadata_prefix) + metadata.each do |key, value| + attrs["#{prefix}.#{key}"] = value.to_s unless value.nil? + end + end + + def build_response_attributes(response) + attrs = {} + attrs['gen_ai.response.model'] = response.model_id if response.model_id + attrs['gen_ai.usage.input_tokens'] = response.input_tokens if response.input_tokens + attrs['gen_ai.usage.output_tokens'] = response.output_tokens if response.output_tokens + attrs + end + + def build_tool_attributes(tool_call:, session_id:) + { + 'langsmith.span.kind' => 'TOOL', + 'gen_ai.operation.name' => 'tool', + 'gen_ai.tool.name' => tool_call.name.to_s, + 'gen_ai.tool.call.id' => tool_call.id, + 'gen_ai.conversation.id' => session_id + } + end + + def build_tool_input_attributes(tool_call:, max_length:) + args = tool_call.arguments + input = args.is_a?(String) ? args : args.to_json + truncated = truncate_content(input, max_length) + { + 'input.value' => truncated, # LangSmith Input panel + 'gen_ai.prompt' => truncated # GenAI convention fallback + } + end + + def build_tool_output_attributes(result:, max_length:) + output = result.is_a?(String) ? result : result.to_s + truncated = truncate_content(output, max_length) + { + 'output.value' => truncated, # LangSmith Output panel + 'gen_ai.completion' => truncated # GenAI convention fallback + } + end + end + end + end +end diff --git a/lib/ruby_llm/tool.rb b/lib/ruby_llm/tool.rb index 2846a995d..24d2468c7 100644 --- a/lib/ruby_llm/tool.rb +++ b/lib/ruby_llm/tool.rb @@ -186,7 +186,7 @@ def resolve_schema def resolve_direct_schema(schema) return extract_schema(schema.to_json_schema) if schema.respond_to?(:to_json_schema) return RubyLLM::Utils.deep_dup(schema) if schema.is_a?(Hash) - if schema.is_a?(Class) && schema.instance_methods.include?(:to_json_schema) + if schema.is_a?(Class) && schema.method_defined?(:to_json_schema) return extract_schema(schema.new.to_json_schema) end diff --git a/spec/fixtures/vcr_cassettes/instrumentation_creates_spans_for_chat.yml b/spec/fixtures/vcr_cassettes/instrumentation_creates_spans_for_chat.yml new file mode 100644 index 000000000..89eb65215 --- /dev/null +++ b/spec/fixtures/vcr_cassettes/instrumentation_creates_spans_for_chat.yml @@ -0,0 +1,116 @@ +--- +http_interactions: +- request: + method: post + uri: https://api.openai.com/v1/chat/completions + body: + encoding: UTF-8 + string: '{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"Hello"}],"stream":false}' + headers: + User-Agent: + - Faraday v2.14.0 + Authorization: + - Bearer + Content-Type: + - application/json + Accept-Encoding: + - gzip;q=1.0,deflate;q=0.6,identity;q=0.3 + Accept: + - "*/*" + response: + status: + code: 200 + message: OK + headers: + Date: + - Fri, 05 Dec 2025 20:43:10 GMT + Content-Type: + - application/json + Transfer-Encoding: + - chunked + Connection: + - keep-alive + Access-Control-Expose-Headers: + - X-Request-ID + Openai-Organization: + - "" + Openai-Processing-Ms: + - '210' + Openai-Project: + - proj_NKne139JTtyy43xNywVdTYx1 + Openai-Version: + - '2020-10-01' + X-Envoy-Upstream-Service-Time: + - '366' + X-Ratelimit-Limit-Requests: + - '30000' + X-Ratelimit-Limit-Tokens: + - '150000000' + X-Ratelimit-Remaining-Requests: + - '29999' + X-Ratelimit-Remaining-Tokens: + - '149999995' + X-Ratelimit-Reset-Requests: + - 2ms + X-Ratelimit-Reset-Tokens: + - 0s + X-Request-Id: + - "" + X-Openai-Proxy-Wasm: + - v0.1 + Cf-Cache-Status: + - DYNAMIC + Set-Cookie: + - "" + - "" + Strict-Transport-Security: + - max-age=31536000; includeSubDomains; preload + X-Content-Type-Options: + - nosniff + Server: + - cloudflare + Cf-Ray: + - "" + Alt-Svc: + - h3=":443"; ma=86400 + body: + encoding: ASCII-8BIT + string: | + { + "id": "chatcmpl-CjWvuw5taodyXNSbbwC53IudDDQVq", + "object": "chat.completion", + "created": 1764967390, + "model": "gpt-4.1-nano-2025-04-14", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Hello! How can I assist you today?", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 8, + "completion_tokens": 9, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_ef015fa747" + } + recorded_at: Fri, 05 Dec 2025 20:43:10 GMT +recorded_with: VCR 6.3.1 diff --git a/spec/fixtures/vcr_cassettes/instrumentation_creates_tool_spans.yml b/spec/fixtures/vcr_cassettes/instrumentation_creates_tool_spans.yml new file mode 100644 index 000000000..50803c053 --- /dev/null +++ b/spec/fixtures/vcr_cassettes/instrumentation_creates_tool_spans.yml @@ -0,0 +1,213 @@ +--- +http_interactions: +- request: + method: post + uri: https://api.openai.com/v1/chat/completions + body: + encoding: UTF-8 + string: '{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"What''s + the weather in Berlin? (52.5200, 13.4050)"}],"stream":false,"tools":[{"type":"function","function":{"name":"weather","description":"Gets + current weather for a location","parameters":{"type":"object","properties":{"latitude":{"type":"string","description":"Latitude + (e.g., 52.5200)"},"longitude":{"type":"string","description":"Longitude (e.g., + 13.4050)"}},"required":["latitude","longitude"],"additionalProperties":false,"strict":true}}}]}' + headers: + User-Agent: + - Faraday v2.14.0 + Authorization: + - Bearer + Content-Type: + - application/json + Accept-Encoding: + - gzip;q=1.0,deflate;q=0.6,identity;q=0.3 + Accept: + - "*/*" + response: + status: + code: 200 + message: OK + headers: + Date: + - Fri, 05 Dec 2025 20:43:11 GMT + Content-Type: + - application/json + Transfer-Encoding: + - chunked + Connection: + - keep-alive + Access-Control-Expose-Headers: + - X-Request-ID + Openai-Organization: + - "" + Openai-Processing-Ms: + - '623' + Openai-Project: + - proj_NKne139JTtyy43xNywVdTYx1 + Openai-Version: + - '2020-10-01' + X-Envoy-Upstream-Service-Time: + - '711' + X-Ratelimit-Limit-Requests: + - '30000' + X-Ratelimit-Limit-Tokens: + - '150000000' + X-Ratelimit-Remaining-Requests: + - '29999' + X-Ratelimit-Remaining-Tokens: + - '149999985' + X-Ratelimit-Reset-Requests: + - 2ms + X-Ratelimit-Reset-Tokens: + - 0s + X-Request-Id: + - "" + X-Openai-Proxy-Wasm: + - v0.1 + Cf-Cache-Status: + - DYNAMIC + Set-Cookie: + - "" + - "" + Strict-Transport-Security: + - max-age=31536000; includeSubDomains; preload + X-Content-Type-Options: + - nosniff + Server: + - cloudflare + Cf-Ray: + - "" + Alt-Svc: + - h3=":443"; ma=86400 + body: + encoding: ASCII-8BIT + string: | + { + "id": "chatcmpl-CjWvvIyt3nRejD0oyQIfdc93V9WvT", + "object": "chat.completion", + "created": 1764967391, + "model": "gpt-4.1-nano-2025-04-14", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_adL6PJ7P2FJ8Aci4Mg0c4HIp", + "type": "function", + "function": { + "name": "weather", + "arguments": "{\"latitude\": \"52.5200\", \"longitude\": \"13.4050\"}" + } + } + ], + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 88, + "completion_tokens": 39, + "total_tokens": 127, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_1a97b5aa6c" + } + recorded_at: Fri, 05 Dec 2025 20:43:11 GMT +- request: + method: post + uri: https://api.openai.com/v1/chat/completions + body: + encoding: UTF-8 + string: '{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"What''s + the weather in Berlin? (52.5200, 13.4050)"},{"role":"assistant","tool_calls":[{"id":"call_adL6PJ7P2FJ8Aci4Mg0c4HIp","type":"function","function":{"name":"weather","arguments":"{\"latitude\":\"52.5200\",\"longitude\":\"13.4050\"}"}}]},{"role":"tool","content":"Current + weather at 52.5200, 13.4050: 15°C, Wind: 10 km/h","tool_call_id":"call_adL6PJ7P2FJ8Aci4Mg0c4HIp"}],"stream":false,"tools":[{"type":"function","function":{"name":"weather","description":"Gets + current weather for a location","parameters":{"type":"object","properties":{"latitude":{"type":"string","description":"Latitude + (e.g., 52.5200)"},"longitude":{"type":"string","description":"Longitude (e.g., + 13.4050)"}},"required":["latitude","longitude"],"additionalProperties":false,"strict":true}}}]}' + headers: + User-Agent: + - Faraday v2.14.0 + Authorization: + - Bearer + Content-Type: + - application/json + Accept-Encoding: + - gzip;q=1.0,deflate;q=0.6,identity;q=0.3 + Accept: + - "*/*" + response: + status: + code: 200 + message: OK + headers: + Date: + - Fri, 05 Dec 2025 20:43:12 GMT + Content-Type: + - application/json + Transfer-Encoding: + - chunked + Connection: + - keep-alive + Access-Control-Expose-Headers: + - X-Request-ID + Openai-Organization: + - "" + Openai-Processing-Ms: + - '425' + Openai-Project: + - proj_NKne139JTtyy43xNywVdTYx1 + Openai-Version: + - '2020-10-01' + X-Envoy-Upstream-Service-Time: + - '532' + X-Ratelimit-Limit-Requests: + - '30000' + X-Ratelimit-Limit-Tokens: + - '150000000' + X-Ratelimit-Remaining-Requests: + - '29999' + X-Ratelimit-Remaining-Tokens: + - '149999967' + X-Ratelimit-Reset-Requests: + - 2ms + X-Ratelimit-Reset-Tokens: + - 0s + X-Request-Id: + - "" + X-Openai-Proxy-Wasm: + - v0.1 + Cf-Cache-Status: + - DYNAMIC + Set-Cookie: + - "" + - "" + Strict-Transport-Security: + - max-age=31536000; includeSubDomains; preload + X-Content-Type-Options: + - nosniff + Server: + - cloudflare + Cf-Ray: + - "" + Alt-Svc: + - h3=":443"; ma=86400 + body: + encoding: ASCII-8BIT + string: !binary |- + ewogICJpZCI6ICJjaGF0Y21wbC1Dald2d0Z6NHFKMjNGNlFKT24zanlkWXFOaTRRQiIsCiAgIm9iamVjdCI6ICJjaGF0LmNvbXBsZXRpb24iLAogICJjcmVhdGVkIjogMTc2NDk2NzM5MiwKICAibW9kZWwiOiAiZ3B0LTQuMS1uYW5vLTIwMjUtMDQtMTQiLAogICJjaG9pY2VzIjogWwogICAgewogICAgICAiaW5kZXgiOiAwLAogICAgICAibWVzc2FnZSI6IHsKICAgICAgICAicm9sZSI6ICJhc3Npc3RhbnQiLAogICAgICAgICJjb250ZW50IjogIlRoZSBjdXJyZW50IHdlYXRoZXIgaW4gQmVybGluIGlzIDE1wrBDIHdpdGggYSB3aW5kIHNwZWVkIG9mIDEwIGttL2guIiwKICAgICAgICAicmVmdXNhbCI6IG51bGwsCiAgICAgICAgImFubm90YXRpb25zIjogW10KICAgICAgfSwKICAgICAgImxvZ3Byb2JzIjogbnVsbCwKICAgICAgImZpbmlzaF9yZWFzb24iOiAic3RvcCIKICAgIH0KICBdLAogICJ1c2FnZSI6IHsKICAgICJwcm9tcHRfdG9rZW5zIjogMTQzLAogICAgImNvbXBsZXRpb25fdG9rZW5zIjogMjAsCiAgICAidG90YWxfdG9rZW5zIjogMTYzLAogICAgInByb21wdF90b2tlbnNfZGV0YWlscyI6IHsKICAgICAgImNhY2hlZF90b2tlbnMiOiAwLAogICAgICAiYXVkaW9fdG9rZW5zIjogMAogICAgfSwKICAgICJjb21wbGV0aW9uX3Rva2Vuc19kZXRhaWxzIjogewogICAgICAicmVhc29uaW5nX3Rva2VucyI6IDAsCiAgICAgICJhdWRpb190b2tlbnMiOiAwLAogICAgICAiYWNjZXB0ZWRfcHJlZGljdGlvbl90b2tlbnMiOiAwLAogICAgICAicmVqZWN0ZWRfcHJlZGljdGlvbl90b2tlbnMiOiAwCiAgICB9CiAgfSwKICAic2VydmljZV90aWVyIjogImRlZmF1bHQiLAogICJzeXN0ZW1fZmluZ2VycHJpbnQiOiAiZnBfMWE5N2I1YWE2YyIKfQo= + recorded_at: Fri, 05 Dec 2025 20:43:12 GMT +recorded_with: VCR 6.3.1 diff --git a/spec/fixtures/vcr_cassettes/instrumentation_includes_token_usage.yml b/spec/fixtures/vcr_cassettes/instrumentation_includes_token_usage.yml new file mode 100644 index 000000000..3017b4813 --- /dev/null +++ b/spec/fixtures/vcr_cassettes/instrumentation_includes_token_usage.yml @@ -0,0 +1,116 @@ +--- +http_interactions: +- request: + method: post + uri: https://api.openai.com/v1/chat/completions + body: + encoding: UTF-8 + string: '{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"Hello"}],"stream":false}' + headers: + User-Agent: + - Faraday v2.14.0 + Authorization: + - Bearer + Content-Type: + - application/json + Accept-Encoding: + - gzip;q=1.0,deflate;q=0.6,identity;q=0.3 + Accept: + - "*/*" + response: + status: + code: 200 + message: OK + headers: + Date: + - Fri, 05 Dec 2025 20:43:13 GMT + Content-Type: + - application/json + Transfer-Encoding: + - chunked + Connection: + - keep-alive + Access-Control-Expose-Headers: + - X-Request-ID + Openai-Organization: + - "" + Openai-Processing-Ms: + - '272' + Openai-Project: + - proj_NKne139JTtyy43xNywVdTYx1 + Openai-Version: + - '2020-10-01' + X-Envoy-Upstream-Service-Time: + - '375' + X-Ratelimit-Limit-Requests: + - '30000' + X-Ratelimit-Limit-Tokens: + - '150000000' + X-Ratelimit-Remaining-Requests: + - '29999' + X-Ratelimit-Remaining-Tokens: + - '149999995' + X-Ratelimit-Reset-Requests: + - 2ms + X-Ratelimit-Reset-Tokens: + - 0s + X-Request-Id: + - "" + X-Openai-Proxy-Wasm: + - v0.1 + Cf-Cache-Status: + - DYNAMIC + Set-Cookie: + - "" + - "" + Strict-Transport-Security: + - max-age=31536000; includeSubDomains; preload + X-Content-Type-Options: + - nosniff + Server: + - cloudflare + Cf-Ray: + - "" + Alt-Svc: + - h3=":443"; ma=86400 + body: + encoding: ASCII-8BIT + string: | + { + "id": "chatcmpl-CjWvxT11aJ9cmSH46FoIZ9X4wrKbb", + "object": "chat.completion", + "created": 1764967393, + "model": "gpt-4.1-nano-2025-04-14", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Hello! How can I assist you today?", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 8, + "completion_tokens": 9, + "total_tokens": 17, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_ef015fa747" + } + recorded_at: Fri, 05 Dec 2025 20:43:13 GMT +recorded_with: VCR 6.3.1 diff --git a/spec/fixtures/vcr_cassettes/instrumentation_maintains_session_id.yml b/spec/fixtures/vcr_cassettes/instrumentation_maintains_session_id.yml new file mode 100644 index 000000000..fa18f2624 --- /dev/null +++ b/spec/fixtures/vcr_cassettes/instrumentation_maintains_session_id.yml @@ -0,0 +1,198 @@ +--- +http_interactions: +- request: + method: post + uri: https://api.openai.com/v1/chat/completions + body: + encoding: UTF-8 + string: '{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"What''s + your favorite color?"}],"stream":false}' + headers: + User-Agent: + - Faraday v2.14.0 + Authorization: + - Bearer + Content-Type: + - application/json + Accept-Encoding: + - gzip;q=1.0,deflate;q=0.6,identity;q=0.3 + Accept: + - "*/*" + response: + status: + code: 200 + message: OK + headers: + Date: + - Fri, 05 Dec 2025 20:43:14 GMT + Content-Type: + - application/json + Transfer-Encoding: + - chunked + Connection: + - keep-alive + Access-Control-Expose-Headers: + - X-Request-ID + Openai-Organization: + - "" + Openai-Processing-Ms: + - '521' + Openai-Project: + - proj_NKne139JTtyy43xNywVdTYx1 + Openai-Version: + - '2020-10-01' + X-Envoy-Upstream-Service-Time: + - '613' + X-Ratelimit-Limit-Requests: + - '30000' + X-Ratelimit-Limit-Tokens: + - '150000000' + X-Ratelimit-Remaining-Requests: + - '29999' + X-Ratelimit-Remaining-Tokens: + - '149999990' + X-Ratelimit-Reset-Requests: + - 2ms + X-Ratelimit-Reset-Tokens: + - 0s + X-Request-Id: + - "" + X-Openai-Proxy-Wasm: + - v0.1 + Cf-Cache-Status: + - DYNAMIC + Set-Cookie: + - "" + - "" + Strict-Transport-Security: + - max-age=31536000; includeSubDomains; preload + X-Content-Type-Options: + - nosniff + Server: + - cloudflare + Cf-Ray: + - "" + Alt-Svc: + - h3=":443"; ma=86400 + body: + encoding: ASCII-8BIT + string: | + { + "id": "chatcmpl-CjWvxIL5R4NWCVmGoornkRReaUDZH", + "object": "chat.completion", + "created": 1764967393, + "model": "gpt-4.1-nano-2025-04-14", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "I don't have personal feelings or preferences, but I think the color blue is quite popular and calming! Do you have a favorite color?", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 12, + "completion_tokens": 27, + "total_tokens": 39, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_ef015fa747" + } + recorded_at: Fri, 05 Dec 2025 20:43:14 GMT +- request: + method: post + uri: https://api.openai.com/v1/chat/completions + body: + encoding: UTF-8 + string: '{"model":"gpt-4.1-nano","messages":[{"role":"user","content":"What''s + your favorite color?"},{"role":"assistant","content":"I don''t have personal + feelings or preferences, but I think the color blue is quite popular and calming! + Do you have a favorite color?"},{"role":"user","content":"Why is that?"}],"stream":false}' + headers: + User-Agent: + - Faraday v2.14.0 + Authorization: + - Bearer + Content-Type: + - application/json + Accept-Encoding: + - gzip;q=1.0,deflate;q=0.6,identity;q=0.3 + Accept: + - "*/*" + response: + status: + code: 200 + message: OK + headers: + Date: + - Fri, 05 Dec 2025 20:43:15 GMT + Content-Type: + - application/json + Transfer-Encoding: + - chunked + Connection: + - keep-alive + Access-Control-Expose-Headers: + - X-Request-ID + Openai-Organization: + - "" + Openai-Processing-Ms: + - '855' + Openai-Project: + - proj_NKne139JTtyy43xNywVdTYx1 + Openai-Version: + - '2020-10-01' + X-Envoy-Upstream-Service-Time: + - '995' + X-Ratelimit-Limit-Requests: + - '30000' + X-Ratelimit-Limit-Tokens: + - '150000000' + X-Ratelimit-Remaining-Requests: + - '29999' + X-Ratelimit-Remaining-Tokens: + - '149999952' + X-Ratelimit-Reset-Requests: + - 2ms + X-Ratelimit-Reset-Tokens: + - 0s + X-Request-Id: + - "" + X-Openai-Proxy-Wasm: + - v0.1 + Cf-Cache-Status: + - DYNAMIC + Set-Cookie: + - "" + - "" + Strict-Transport-Security: + - max-age=31536000; includeSubDomains; preload + X-Content-Type-Options: + - nosniff + Server: + - cloudflare + Cf-Ray: + - "" + Alt-Svc: + - h3=":443"; ma=86400 + body: + encoding: ASCII-8BIT + string: !binary |- + ewogICJpZCI6ICJjaGF0Y21wbC1Dald2emZnM0dqaWxsdnpId3U1R3VDZnZKeXBTMCIsCiAgIm9iamVjdCI6ICJjaGF0LmNvbXBsZXRpb24iLAogICJjcmVhdGVkIjogMTc2NDk2NzM5NSwKICAibW9kZWwiOiAiZ3B0LTQuMS1uYW5vLTIwMjUtMDQtMTQiLAogICJjaG9pY2VzIjogWwogICAgewogICAgICAiaW5kZXgiOiAwLAogICAgICAibWVzc2FnZSI6IHsKICAgICAgICAicm9sZSI6ICJhc3Npc3RhbnQiLAogICAgICAgICJjb250ZW50IjogIlBlb3BsZSBvZnRlbiBmYXZvciBibHVlIGJlY2F1c2UgaXQncyBhc3NvY2lhdGVkIHdpdGggcXVhbGl0aWVzIGxpa2UgY2FsbW5lc3MsIHRydXN0LCBhbmQgdHJhbnF1aWxpdHkuIEl0J3MgYSBjb2xvciBjb21tb25seSBmb3VuZCBpbiBuYXR1cmXigJR0aGluayB0aGUgc2t5IGFuZCB0aGUgb2NlYW7igJR3aGljaCBjYW4gZXZva2UgZmVlbGluZ3Mgb2YgcGVhY2UgYW5kIHJlbGF4YXRpb24uIEFkZGl0aW9uYWxseSwgbWFueSBjdWx0dXJlcyBzZWUgYmx1ZSBhcyBhIHN0YWJsZSBhbmQgZGVwZW5kYWJsZSBjb2xvciwgd2hpY2ggbWlnaHQgY29udHJpYnV0ZSB0byBpdHMgcG9wdWxhcml0eS4gRG8geW91IGhhdmUgYSByZWFzb24gd2h5IHlvdSBsaWtlIGEgcGFydGljdWxhciBjb2xvcj8iLAogICAgICAgICJyZWZ1c2FsIjogbnVsbCwKICAgICAgICAiYW5ub3RhdGlvbnMiOiBbXQogICAgICB9LAogICAgICAibG9ncHJvYnMiOiBudWxsLAogICAgICAiZmluaXNoX3JlYXNvbiI6ICJzdG9wIgogICAgfQogIF0sCiAgInVzYWdlIjogewogICAgInByb21wdF90b2tlbnMiOiA1MSwKICAgICJjb21wbGV0aW9uX3Rva2VucyI6IDczLAogICAgInRvdGFsX3Rva2VucyI6IDEyNCwKICAgICJwcm9tcHRfdG9rZW5zX2RldGFpbHMiOiB7CiAgICAgICJjYWNoZWRfdG9rZW5zIjogMCwKICAgICAgImF1ZGlvX3Rva2VucyI6IDAKICAgIH0sCiAgICAiY29tcGxldGlvbl90b2tlbnNfZGV0YWlscyI6IHsKICAgICAgInJlYXNvbmluZ190b2tlbnMiOiAwLAogICAgICAiYXVkaW9fdG9rZW5zIjogMCwKICAgICAgImFjY2VwdGVkX3ByZWRpY3Rpb25fdG9rZW5zIjogMCwKICAgICAgInJlamVjdGVkX3ByZWRpY3Rpb25fdG9rZW5zIjogMAogICAgfQogIH0sCiAgInNlcnZpY2VfdGllciI6ICJkZWZhdWx0IiwKICAic3lzdGVtX2ZpbmdlcnByaW50IjogImZwX2VmMDE1ZmE3NDciCn0K + recorded_at: Fri, 05 Dec 2025 20:43:15 GMT +recorded_with: VCR 6.3.1 diff --git a/spec/ruby_llm/active_record/acts_as_model_spec.rb b/spec/ruby_llm/active_record/acts_as_model_spec.rb index 56fb01570..5534baa85 100644 --- a/spec/ruby_llm/active_record/acts_as_model_spec.rb +++ b/spec/ruby_llm/active_record/acts_as_model_spec.rb @@ -255,7 +255,8 @@ def messages # Mock the chat creation to verify parameters expect(RubyLLM).to receive(:chat).with( # rubocop:disable RSpec/MessageSpies,RSpec/StubbedMock model: 'test-gpt', - provider: :openai + provider: :openai, + session_id: chat.id ).and_return( instance_double(RubyLLM::Chat, reset_messages!: nil, add_message: nil, instance_variable_get: {}, on_new_message: nil, on_end_message: nil, @@ -272,7 +273,8 @@ def messages expect(RubyLLM).to receive(:chat).with( # rubocop:disable RSpec/MessageSpies,RSpec/StubbedMock model: 'test-claude', - provider: :anthropic + provider: :anthropic, + session_id: chat.id ).and_return( instance_double(RubyLLM::Chat, reset_messages!: nil, add_message: nil, instance_variable_get: {}, on_new_message: nil, on_end_message: nil, diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb new file mode 100644 index 000000000..7f497d3f3 --- /dev/null +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -0,0 +1,541 @@ +# frozen_string_literal: true + +require 'opentelemetry/sdk' + +RSpec.describe RubyLLM::Instrumentation do + describe 'Configuration' do + it 'has tracing_enabled defaulting to false' do + config = RubyLLM::Configuration.new + expect(config.tracing_enabled).to be false + end + + it 'has tracing_log_content defaulting to false' do + config = RubyLLM::Configuration.new + expect(config.tracing_log_content).to be false + end + + it 'has tracing_max_content_length defaulting to 10000' do + config = RubyLLM::Configuration.new + expect(config.tracing_max_content_length).to eq 10_000 + end + + it 'has tracing_metadata_prefix defaulting to metadata' do + config = RubyLLM::Configuration.new + expect(config.tracing_metadata_prefix).to eq 'metadata' + end + + it 'allows configuration via block' do + RubyLLM.configure do |config| + config.tracing_enabled = true + config.tracing_log_content = true + config.tracing_max_content_length = 5000 + config.tracing_metadata_prefix = 'app.metadata' + end + + expect(RubyLLM.config.tracing_enabled).to be true + expect(RubyLLM.config.tracing_log_content).to be true + expect(RubyLLM.config.tracing_max_content_length).to eq 5000 + expect(RubyLLM.config.tracing_metadata_prefix).to eq 'app.metadata' + end + end + + describe 'RubyLLM::Instrumentation' do + describe '.enabled?' do + it 'returns false when tracing_enabled is false' do + RubyLLM.configure { |c| c.tracing_enabled = false } + expect(described_class.enabled?).to be false + end + + it 'returns true when tracing_enabled is true and OpenTelemetry is available' do + RubyLLM.configure { |c| c.tracing_enabled = true } + described_class.reset! + expect(described_class.enabled?).to be true + end + + it 'raises an error when tracing_enabled is true but OpenTelemetry is not available' do + RubyLLM.configure { |c| c.tracing_enabled = true } + allow(described_class).to receive(:otel_available?).and_return(false) + + expect { described_class.enabled? }.to raise_error( + RubyLLM::ConfigurationError, + /OpenTelemetry is not available/ + ) + end + end + + describe '.tracer' do + it 'returns NullTracer when disabled' do + RubyLLM.configure { |c| c.tracing_enabled = false } + expect(described_class.tracer).to be_a(RubyLLM::Instrumentation::NullTracer) + end + end + end + + describe 'RubyLLM::Instrumentation::NullTracer' do + let(:tracer) { RubyLLM::Instrumentation::NullTracer.instance } + + it 'is a singleton' do + expect(tracer).to be RubyLLM::Instrumentation::NullTracer.instance + end + + describe '#in_span' do + it 'yields a NullSpan' do + tracer.in_span('test.span') do |span| + expect(span).to be_a(RubyLLM::Instrumentation::NullSpan) + end + end + + it 'returns the block result' do + result = tracer.in_span('test.span') { 'hello' } + expect(result).to eq 'hello' + end + + it 'propagates exceptions' do + expect do + tracer.in_span('test.span') { raise 'boom' } + end.to raise_error('boom') + end + end + end + + describe 'RubyLLM::Instrumentation::NullSpan' do + let(:span) { RubyLLM::Instrumentation::NullSpan.instance } + + it 'is a singleton' do + expect(span).to be RubyLLM::Instrumentation::NullSpan.instance + end + + it 'responds to recording? and returns false' do + expect(span.recording?).to be false + end + + it 'responds to set_attribute and returns self' do + expect(span.set_attribute('key', 'value')).to be span + end + + it 'responds to add_attributes and returns self' do + expect(span.add_attributes(key: 'value')).to be span + end + + it 'responds to record_exception and returns self' do + expect(span.record_exception(StandardError.new)).to be span + end + + it 'responds to status= and returns nil' do + expect(span.status = 'error').to eq 'error' + end + end + + describe 'Chat#session_id' do + include_context 'with configured RubyLLM' + + it 'generates a unique session_id for each Chat instance' do + chat1 = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai) + chat2 = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai) + + expect(chat1.session_id).to be_a(String) + expect(chat1.session_id).to match(/\A[0-9a-f-]{36}\z/) # UUID format + expect(chat1.session_id).not_to eq(chat2.session_id) + end + + it 'maintains the same session_id across multiple asks' do + chat = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai) + session_id = chat.session_id + + # session_id should remain constant + expect(chat.session_id).to eq(session_id) + expect(chat.session_id).to eq(session_id) + end + + it 'accepts a custom session_id' do + custom_id = 'my-conversation-123' + chat = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai, + session_id: custom_id) + + expect(chat.session_id).to eq(custom_id) + end + + it 'generates a UUID when session_id is nil' do + chat = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai, session_id: nil) + + expect(chat.session_id).to be_a(String) + expect(chat.session_id).to match(/\A[0-9a-f-]{36}\z/) + end + end + + describe 'RubyLLM::Instrumentation::SpanBuilder' do + describe '.build_tool_attributes' do + it 'builds attributes for a tool call' do + tool_call = instance_double(RubyLLM::ToolCall, name: 'get_weather', id: 'call_123', + arguments: { location: 'NYC' }) + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_tool_attributes( + tool_call: tool_call, + session_id: 'session-abc' + ) + + expect(attrs['gen_ai.operation.name']).to eq 'tool' + expect(attrs['gen_ai.tool.name']).to eq 'get_weather' + expect(attrs['gen_ai.tool.call.id']).to eq 'call_123' + expect(attrs['gen_ai.conversation.id']).to eq 'session-abc' + end + end + + describe '.build_tool_input_attributes' do + it 'builds input attributes with truncation' do + tool_call = instance_double(RubyLLM::ToolCall, arguments: { query: 'a' * 100 }) + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_tool_input_attributes( + tool_call: tool_call, + max_length: 50 + ) + + expect(attrs['input.value']).to include('[truncated]') + end + end + + describe '.build_tool_output_attributes' do + it 'builds output attributes with truncation' do + result = 'b' * 100 + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_tool_output_attributes( + result: result, + max_length: 50 + ) + + expect(attrs['output.value']).to include('[truncated]') + end + end + + describe '.truncate_content' do + it 'returns content as-is when under max length' do + content = 'short content' + result = RubyLLM::Instrumentation::SpanBuilder.truncate_content(content, 100) + expect(result).to eq 'short content' + end + + it 'truncates content when over max length' do + content = 'a' * 100 + result = RubyLLM::Instrumentation::SpanBuilder.truncate_content(content, 50) + expect(result).to eq "#{'a' * 50}... [truncated]" + end + + it 'handles nil content' do + result = RubyLLM::Instrumentation::SpanBuilder.truncate_content(nil, 100) + expect(result).to be_nil + end + end + + describe '.extract_content_text' do + it 'returns string content as-is' do + result = RubyLLM::Instrumentation::SpanBuilder.extract_content_text('Hello') + expect(result).to eq 'Hello' + end + + it 'extracts text from Content objects' do + content = RubyLLM::Content.new('Hello with attachment', ['spec/fixtures/ruby.png']) + result = RubyLLM::Instrumentation::SpanBuilder.extract_content_text(content) + expect(result).to eq 'Hello with attachment' + end + + it 'describes attachments for Content objects without text' do + content = RubyLLM::Content.new(nil, ['spec/fixtures/ruby.png']) + result = RubyLLM::Instrumentation::SpanBuilder.extract_content_text(content) + expect(result).to eq '[image: ruby.png]' + end + + it 'describes multiple attachments' do + content = RubyLLM::Content.new(nil, ['spec/fixtures/ruby.png', 'spec/fixtures/ruby.mp3']) + result = RubyLLM::Instrumentation::SpanBuilder.extract_content_text(content) + expect(result).to eq '[image: ruby.png, audio: ruby.mp3]' + end + end + + describe '.build_message_attributes' do + it 'builds indexed attributes for messages' do + messages = [ + RubyLLM::Message.new(role: :system, content: 'You are helpful'), + RubyLLM::Message.new(role: :user, content: 'Hello') + ] + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 1000) + + expect(attrs['gen_ai.prompt.0.role']).to eq 'system' + expect(attrs['gen_ai.prompt.0.content']).to eq 'You are helpful' + expect(attrs['gen_ai.prompt.1.role']).to eq 'user' + expect(attrs['gen_ai.prompt.1.content']).to eq 'Hello' + end + + it 'truncates long content' do + messages = [RubyLLM::Message.new(role: :user, content: 'a' * 100)] + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 50) + + expect(attrs['gen_ai.prompt.0.content']).to eq "#{'a' * 50}... [truncated]" + end + + it 'handles Content objects with attachments' do + content = RubyLLM::Content.new('Describe this image', ['spec/fixtures/ruby.png']) + messages = [RubyLLM::Message.new(role: :user, content: content)] + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 1000) + + expect(attrs['gen_ai.prompt.0.content']).to eq 'Describe this image' + end + end + + describe '.build_metadata_attributes' do + it 'builds attributes with the given prefix' do + attrs = {} + RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( + attrs, + { user_id: 123, request_id: 'abc' }, + prefix: 'langsmith.metadata' + ) + + expect(attrs['langsmith.metadata.user_id']).to eq '123' + expect(attrs['langsmith.metadata.request_id']).to eq 'abc' + end + + it 'supports custom prefixes' do + attrs = {} + RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( + attrs, + { user_id: 123 }, + prefix: 'app.metadata' + ) + + expect(attrs['app.metadata.user_id']).to eq '123' + end + + it 'skips nil values' do + attrs = {} + RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( + attrs, + { user_id: 123, empty: nil }, + prefix: 'test' + ) + + expect(attrs).to have_key('test.user_id') + expect(attrs).not_to have_key('test.empty') + end + end + end + + describe 'Sampling-aware behavior' do + include_context 'with configured RubyLLM' + + let(:non_recording_span) do + instance_double( + OpenTelemetry::Trace::Span, + recording?: false, + add_attributes: nil, + set_attribute: nil, + record_exception: nil, + 'status=': nil + ) + end + + let(:recording_span) do + instance_double( + OpenTelemetry::Trace::Span, + recording?: true, + add_attributes: nil, + set_attribute: nil, + record_exception: nil, + 'status=': nil + ) + end + + let(:mock_tracer) do + tracer = instance_double(OpenTelemetry::Trace::Tracer) + allow(tracer).to receive(:in_span).and_yield(non_recording_span).and_return(nil) + tracer + end + + before do + RubyLLM.configure do |config| + config.tracing_enabled = true + config.tracing_log_content = true + end + allow(described_class).to receive(:tracer).and_return(mock_tracer) + end + + after do + RubyLLM.configure do |config| + config.tracing_enabled = false + config.tracing_log_content = false + end + described_class.reset! + end + + it 'skips attribute building when span is not recording (sampled out)' do + chat = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai) + + # Mock provider to return a response + mock_response = RubyLLM::Message.new( + role: :assistant, + content: 'Hello!', + input_tokens: 10, + output_tokens: 5, + model_id: 'gpt-4o-mini' + ) + allow(chat.instance_variable_get(:@provider)).to receive(:complete).and_return(mock_response) + + chat.ask('Hello') + + # When recording? is false, add_attributes should NOT be called + expect(non_recording_span).not_to have_received(:add_attributes) + end + + it 'adds attributes when span is recording' do + allow(mock_tracer).to receive(:in_span).and_yield(recording_span).and_return(nil) + + chat = RubyLLM::Chat.new(model: 'gpt-4o-mini', assume_model_exists: true, provider: :openai) + + mock_response = RubyLLM::Message.new( + role: :assistant, + content: 'Hello!', + input_tokens: 10, + output_tokens: 5, + model_id: 'gpt-4o-mini' + ) + allow(chat.instance_variable_get(:@provider)).to receive(:complete).and_return(mock_response) + + chat.ask('Hello') + + # When recording? is true, add_attributes should be called + expect(recording_span).to have_received(:add_attributes).at_least(:once) + end + end + + describe 'OpenTelemetry Integration', :vcr do + include_context 'with configured RubyLLM' + + let(:exporter) { OpenTelemetry::SDK::Trace::Export::InMemorySpanExporter.new } + + # Use anonymous class to avoid leaking constants + let(:weather_tool) do + Class.new(RubyLLM::Tool) do + description 'Gets current weather for a location' + param :latitude, desc: 'Latitude (e.g., 52.5200)' + param :longitude, desc: 'Longitude (e.g., 13.4050)' + + def execute(latitude:, longitude:) + "Current weather at #{latitude}, #{longitude}: 15°C, Wind: 10 km/h" + end + + def self.name + 'Weather' + end + end + end + + before do + # Reset any existing OTel configuration + OpenTelemetry::SDK.configure do |c| + c.add_span_processor( + OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(exporter) + ) + end + + # Only configure tracing options - don't overwrite API keys + RubyLLM.configure do |config| + config.tracing_enabled = true + config.tracing_log_content = true + end + + # Reset the cached tracer + described_class.reset! + end + + after do + RubyLLM.configure do |config| + config.tracing_enabled = false + config.tracing_log_content = false + end + described_class.reset! + exporter.reset + end + + it 'creates spans with correct attributes when tracing is enabled' do + VCR.use_cassette('instrumentation_creates_spans_for_chat') do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + chat.ask('Hello') + end + + spans = exporter.finished_spans + expect(spans.count).to be >= 1 + + chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } + expect(chat_span).not_to be_nil + expect(chat_span.kind).to eq(:client) + expect(chat_span.attributes['gen_ai.system']).to eq('openai') + expect(chat_span.attributes['gen_ai.operation.name']).to eq('chat') + expect(chat_span.attributes['gen_ai.conversation.id']).to be_a(String) + expect(chat_span.attributes['gen_ai.prompt.0.role']).to eq('user') + expect(chat_span.attributes['gen_ai.prompt.0.content']).to eq('Hello') + end + + it 'creates tool spans as children when tools are used' do + VCR.use_cassette('instrumentation_creates_tool_spans') do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + chat.with_tool(weather_tool) + chat.ask("What's the weather in Berlin? (52.5200, 13.4050)") + end + + spans = exporter.finished_spans + tool_span = spans.find { |s| s.name == 'ruby_llm.tool' } + + expect(tool_span).not_to be_nil + expect(tool_span.kind).to eq(:internal) + expect(tool_span.attributes['gen_ai.tool.name']).to be_a(String) + end + + it 'includes token usage in spans' do + VCR.use_cassette('instrumentation_includes_token_usage') do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + chat.ask('Hello') + end + + spans = exporter.finished_spans + chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } + + expect(chat_span.attributes['gen_ai.usage.input_tokens']).to be > 0 + expect(chat_span.attributes['gen_ai.usage.output_tokens']).to be > 0 + end + + it 'maintains session_id across multiple asks' do + VCR.use_cassette('instrumentation_maintains_session_id') do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + chat.ask("What's your favorite color?") + chat.ask('Why is that?') + end + + spans = exporter.finished_spans + chat_spans = spans.select { |s| s.name == 'ruby_llm.chat' } + + expect(chat_spans.count).to eq(2) + session_ids = chat_spans.map { |s| s.attributes['gen_ai.conversation.id'] }.uniq + expect(session_ids.count).to eq(1) # Same session_id for both + end + + it 'records errors on spans when API calls fail' do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + + # Stub the provider to raise an error + allow(chat.instance_variable_get(:@provider)).to receive(:complete).and_raise( + RubyLLM::ServerError.new(nil, 'Test API error') + ) + + expect { chat.ask('Hello') }.to raise_error(RubyLLM::ServerError) + + spans = exporter.finished_spans + chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } + + expect(chat_span).not_to be_nil + expect(chat_span.status.code).to eq(OpenTelemetry::Trace::Status::ERROR) + expect(chat_span.events.any? { |e| e.name == 'exception' }).to be true + end + end +end From bae09c72dfeed4cc08de1094a074d888befc9a77 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:23:03 -0800 Subject: [PATCH 02/19] Fix config scoping bug in Instrumentation Instrumentation.enabled? and Instrumentation.tracer now accept an optional config parameter, allowing Chat to pass its context-specific config instead of always using the global RubyLLM.config. This ensures that per-context tracing configuration is respected. --- lib/ruby_llm/chat.rb | 4 ++-- lib/ruby_llm/instrumentation.rb | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index e6c46cb40..36d7b1594 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -132,7 +132,7 @@ def complete(&) # Skip instrumentation for streaming (not supported yet) return complete_without_instrumentation(&) if block_given? - Instrumentation.tracer.in_span('ruby_llm.chat', kind: Instrumentation::SpanKind::CLIENT) do |span| + Instrumentation.tracer(@config).in_span('ruby_llm.chat', kind: Instrumentation::SpanKind::CLIENT) do |span| complete_with_span(span, &) end end @@ -292,7 +292,7 @@ def handle_tool_calls(response, &) # rubocop:disable Metrics/PerceivedComplexity end def execute_tool(tool_call) - Instrumentation.tracer.in_span('ruby_llm.tool', kind: Instrumentation::SpanKind::INTERNAL) do |span| + Instrumentation.tracer(@config).in_span('ruby_llm.tool', kind: Instrumentation::SpanKind::INTERNAL) do |span| execute_tool_with_span(tool_call, span) end end diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index 37729a4dd..430700536 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -12,8 +12,8 @@ module SpanKind INTERNAL = :internal end class << self - def enabled? - return false unless RubyLLM.config.tracing_enabled + def enabled?(config = RubyLLM.config) + return false unless config.tracing_enabled unless otel_available? raise RubyLLM::ConfigurationError, <<~MSG.strip @@ -31,8 +31,8 @@ def enabled? true end - def tracer - return NullTracer.instance unless enabled? + def tracer(config = RubyLLM.config) + return NullTracer.instance unless enabled?(config) @tracer ||= OpenTelemetry.tracer_provider.tracer( 'ruby_llm', From 6353c213bf3399dcc92de73e7c7950cc83cc98dd Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:23:34 -0800 Subject: [PATCH 03/19] Ensure session_id is consistently a string type Convert AR record ID to string when passing to Chat#session_id to ensure type consistency. Chat defaults to UUID (string) when no session_id is provided, so AR adapters should also pass strings. --- lib/ruby_llm/active_record/acts_as_legacy.rb | 6 +++--- lib/ruby_llm/active_record/chat_methods.rb | 2 +- spec/ruby_llm/active_record/acts_as_model_spec.rb | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/ruby_llm/active_record/acts_as_legacy.rb b/lib/ruby_llm/active_record/acts_as_legacy.rb index abc43880d..512b2bdfe 100644 --- a/lib/ruby_llm/active_record/acts_as_legacy.rb +++ b/lib/ruby_llm/active_record/acts_as_legacy.rb @@ -88,11 +88,11 @@ module ChatLegacyMethods def to_llm(context: nil) # model_id is a string that RubyLLM can resolve - # session_id uses the AR record ID for tracing session grouping + # session_id uses the AR record ID (as string) for tracing session grouping @chat ||= if context - context.chat(model: model_id, session_id: id) + context.chat(model: model_id, session_id: id.to_s) else - RubyLLM.chat(model: model_id, session_id: id) + RubyLLM.chat(model: model_id, session_id: id.to_s) end @chat.reset_messages! diff --git a/lib/ruby_llm/active_record/chat_methods.rb b/lib/ruby_llm/active_record/chat_methods.rb index f95269eb5..c8f97f5af 100644 --- a/lib/ruby_llm/active_record/chat_methods.rb +++ b/lib/ruby_llm/active_record/chat_methods.rb @@ -80,7 +80,7 @@ def to_llm @chat ||= (context || RubyLLM).chat( model: model_record.model_id, provider: model_record.provider.to_sym, - session_id: id # Use AR record ID for tracing session grouping + session_id: id.to_s # Use AR record ID for tracing session grouping ) @chat.reset_messages! diff --git a/spec/ruby_llm/active_record/acts_as_model_spec.rb b/spec/ruby_llm/active_record/acts_as_model_spec.rb index 5534baa85..14a62ae2d 100644 --- a/spec/ruby_llm/active_record/acts_as_model_spec.rb +++ b/spec/ruby_llm/active_record/acts_as_model_spec.rb @@ -256,7 +256,7 @@ def messages expect(RubyLLM).to receive(:chat).with( # rubocop:disable RSpec/MessageSpies,RSpec/StubbedMock model: 'test-gpt', provider: :openai, - session_id: chat.id + session_id: chat.id.to_s ).and_return( instance_double(RubyLLM::Chat, reset_messages!: nil, add_message: nil, instance_variable_get: {}, on_new_message: nil, on_end_message: nil, @@ -274,7 +274,7 @@ def messages expect(RubyLLM).to receive(:chat).with( # rubocop:disable RSpec/MessageSpies,RSpec/StubbedMock model: 'test-claude', provider: :anthropic, - session_id: chat.id + session_id: chat.id.to_s ).and_return( instance_double(RubyLLM::Chat, reset_messages!: nil, add_message: nil, instance_variable_get: {}, on_new_message: nil, on_end_message: nil, From cd460bd803d72c874021793cee99d61431b8f164 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:24:06 -0800 Subject: [PATCH 04/19] Preserve native types in metadata attributes OTel supports string, int, float, and bool attribute values natively. Only stringify complex objects that don't fit these types. This improves filtering and querying capabilities in observability backends. --- lib/ruby_llm/instrumentation.rb | 15 ++++++++++++++- spec/ruby_llm/instrumentation_spec.rb | 24 ++++++++++++++++++++---- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index 430700536..ba2a2db63 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -163,7 +163,20 @@ def build_request_attributes(model:, provider:, session_id:, temperature: nil, m def build_metadata_attributes(attrs, metadata, prefix: RubyLLM.config.tracing_metadata_prefix) metadata.each do |key, value| - attrs["#{prefix}.#{key}"] = value.to_s unless value.nil? + next if value.nil? + + # Preserve native types that OTel supports (string, int, float, bool) + # Only stringify complex objects + attrs["#{prefix}.#{key}"] = otel_safe_value(value) + end + end + + def otel_safe_value(value) + case value + when String, Integer, Float, TrueClass, FalseClass + value + else + value.to_s end end diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index 7f497d3f3..3608cc208 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -285,16 +285,18 @@ end describe '.build_metadata_attributes' do - it 'builds attributes with the given prefix' do + it 'builds attributes with the given prefix preserving native types' do attrs = {} RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( attrs, - { user_id: 123, request_id: 'abc' }, + { user_id: 123, request_id: 'abc', active: true, score: 0.95 }, prefix: 'langsmith.metadata' ) - expect(attrs['langsmith.metadata.user_id']).to eq '123' + expect(attrs['langsmith.metadata.user_id']).to eq 123 expect(attrs['langsmith.metadata.request_id']).to eq 'abc' + expect(attrs['langsmith.metadata.active']).to be true + expect(attrs['langsmith.metadata.score']).to eq 0.95 end it 'supports custom prefixes' do @@ -305,7 +307,7 @@ prefix: 'app.metadata' ) - expect(attrs['app.metadata.user_id']).to eq '123' + expect(attrs['app.metadata.user_id']).to eq 123 end it 'skips nil values' do @@ -319,6 +321,20 @@ expect(attrs).to have_key('test.user_id') expect(attrs).not_to have_key('test.empty') end + + it 'stringifies complex objects' do + attrs = {} + complex_obj = Object.new + def complex_obj.to_s = 'complex_value' + + RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( + attrs, + { data: complex_obj }, + prefix: 'test' + ) + + expect(attrs['test.data']).to eq 'complex_value' + end end end From 2884c5cdf2c1fb279b26f7d0d1483d6d09d64842 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:24:21 -0800 Subject: [PATCH 05/19] Return defensive copy of metadata hash Replace attr_reader :metadata with a method that returns @metadata.dup to prevent external mutation of internal state. Callers must use with_metadata to modify the hash. --- lib/ruby_llm/chat.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index 36d7b1594..b395c9d0d 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -5,7 +5,11 @@ module RubyLLM class Chat include Enumerable - attr_reader :model, :messages, :tools, :params, :headers, :schema, :session_id, :metadata + attr_reader :model, :messages, :tools, :params, :headers, :schema, :session_id + + def metadata + @metadata.dup + end def initialize(model: nil, provider: nil, assume_model_exists: false, context: nil, session_id: nil) if assume_model_exists && !provider From 0661647fe9d8d6774a63cedada7cc22e30dd9c72 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:30:41 -0800 Subject: [PATCH 06/19] Gate LangSmith-specific attributes behind tracing_langsmith_compat Add tracing_langsmith_compat config option (default: false) that: - Adds langsmith.span.kind attribute to spans - Adds input.value/output.value for LangSmith panels - Auto-sets tracing_metadata_prefix to 'langsmith.metadata' Standard gen_ai.* attributes are always emitted regardless of this setting. This allows users of other OTel backends to avoid the LangSmith-specific noise while LangSmith users get full integration. Also replaces the made-up gen_ai.prompt/gen_ai.completion fallback attributes with proper gen_ai.tool.input/gen_ai.tool.output for tool spans. --- docs/_advanced/observability.md | 17 +++- lib/ruby_llm/chat.rb | 22 +++-- lib/ruby_llm/configuration.rb | 11 ++- lib/ruby_llm/instrumentation.rb | 52 +++++----- spec/ruby_llm/instrumentation_spec.rb | 137 +++++++++++++++++++++++++- 5 files changed, 201 insertions(+), 38 deletions(-) diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md index 6c3b2726b..0edc4fbdd 100644 --- a/docs/_advanced/observability.md +++ b/docs/_advanced/observability.md @@ -77,11 +77,13 @@ RubyLLM.configure do |config| config.tracing_enabled = true # Log prompt/completion content (default: false) - # Enable for full LangSmith functionality config.tracing_log_content = true # Max content length before truncation (default: 10000) config.tracing_max_content_length = 5000 + + # Enable LangSmith-specific span attributes (default: false) + config.tracing_langsmith_compat = true end ``` @@ -106,11 +108,13 @@ chat = RubyLLM.chat chat.ask("Hello!") ``` -Metadata appears as `metadata.*` attributes by default. For LangSmith's metadata panel, set: +Metadata appears as `metadata.*` attributes by default. When `tracing_langsmith_compat` is enabled, metadata uses the `langsmith.metadata.*` prefix for proper LangSmith panel integration. + +You can also set a custom prefix: ```ruby RubyLLM.configure do |config| - config.tracing_metadata_prefix = 'langsmith.metadata' + config.tracing_metadata_prefix = 'app.metadata' end ``` @@ -127,7 +131,7 @@ LangSmith is LangChain's observability platform with specialized LLM debugging f RubyLLM.configure do |config| config.tracing_enabled = true config.tracing_log_content = true - config.tracing_metadata_prefix = 'langsmith.metadata' + config.tracing_langsmith_compat = true # Adds LangSmith-specific span attributes end ``` @@ -153,6 +157,11 @@ end LangSmith uses the `Langsmith-Project` header (not `service_name`) to organize traces. +When `tracing_langsmith_compat = true`, RubyLLM adds these additional attributes for LangSmith integration: +- `langsmith.span.kind` - Identifies span type (LLM, TOOL) +- `input.value` / `output.value` - Populates LangSmith's Input/Output panels +- `langsmith.metadata.*` - Custom metadata appears in LangSmith's metadata panel + ### Other Backends RubyLLM works with any OpenTelemetry-compatible backend. Configure the `opentelemetry-exporter-otlp` gem to send traces to your platform of choice. diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index b395c9d0d..9d27e1b16 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -159,6 +159,8 @@ def complete_without_instrumentation(&) end def complete_with_span(span, &) + langsmith = @config.tracing_langsmith_compat + # Set request attributes if span.recording? span.add_attributes( @@ -167,7 +169,9 @@ def complete_with_span(span, &) provider: @provider.slug, session_id: @session_id, temperature: @temperature, - metadata: @metadata + metadata: @metadata, + langsmith_compat: langsmith, + metadata_prefix: @config.tracing_metadata_prefix ) ) @@ -176,7 +180,8 @@ def complete_with_span(span, &) span.add_attributes( Instrumentation::SpanBuilder.build_message_attributes( messages, - max_length: @config.tracing_max_content_length + max_length: @config.tracing_max_content_length, + langsmith_compat: langsmith ) ) end @@ -200,7 +205,8 @@ def complete_with_span(span, &) span.add_attributes( Instrumentation::SpanBuilder.build_completion_attributes( response, - max_length: @config.tracing_max_content_length + max_length: @config.tracing_max_content_length, + langsmith_compat: langsmith ) ) end @@ -304,12 +310,14 @@ def execute_tool(tool_call) def execute_tool_with_span(tool_call, span) tool = tools[tool_call.name.to_sym] args = tool_call.arguments + langsmith = @config.tracing_langsmith_compat if span.recording? span.add_attributes( Instrumentation::SpanBuilder.build_tool_attributes( tool_call: tool_call, - session_id: @session_id + session_id: @session_id, + langsmith_compat: langsmith ) ) @@ -317,7 +325,8 @@ def execute_tool_with_span(tool_call, span) span.add_attributes( Instrumentation::SpanBuilder.build_tool_input_attributes( tool_call: tool_call, - max_length: @config.tracing_max_content_length + max_length: @config.tracing_max_content_length, + langsmith_compat: langsmith ) ) end @@ -329,7 +338,8 @@ def execute_tool_with_span(tool_call, span) span.add_attributes( Instrumentation::SpanBuilder.build_tool_output_attributes( result: result, - max_length: @config.tracing_max_content_length + max_length: @config.tracing_max_content_length, + langsmith_compat: langsmith ) ) end diff --git a/lib/ruby_llm/configuration.rb b/lib/ruby_llm/configuration.rb index 4f7218a54..475a09a18 100644 --- a/lib/ruby_llm/configuration.rb +++ b/lib/ruby_llm/configuration.rb @@ -51,7 +51,8 @@ class Configuration :tracing_enabled, :tracing_log_content, :tracing_max_content_length, - :tracing_metadata_prefix + :tracing_metadata_prefix, + :tracing_langsmith_compat def initialize @request_timeout = 300 @@ -79,6 +80,14 @@ def initialize @tracing_log_content = false @tracing_max_content_length = 10_000 @tracing_metadata_prefix = 'metadata' + @tracing_langsmith_compat = false + end + + def tracing_langsmith_compat=(value) + @tracing_langsmith_compat = value + # Auto-set metadata prefix for LangSmith when enabling compat mode, + # but only if the user hasn't customized it + @tracing_metadata_prefix = 'langsmith.metadata' if value && @tracing_metadata_prefix == 'metadata' end def instance_variables diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index ba2a2db63..afabbfd6e 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -121,7 +121,7 @@ def describe_attachments(attachments) "[#{descriptions.join(', ')}]" end - def build_message_attributes(messages, max_length:) + def build_message_attributes(messages, max_length:, langsmith_compat: false) attrs = {} messages.each_with_index do |msg, idx| attrs["gen_ai.prompt.#{idx}.role"] = msg.role.to_s @@ -129,39 +129,42 @@ def build_message_attributes(messages, max_length:) attrs["gen_ai.prompt.#{idx}.content"] = truncate_content(content, max_length) end # Set input.value for LangSmith Input panel (last user message) - last_user_msg = messages.reverse.find { |m| m.role.to_s == 'user' } - if last_user_msg - content = extract_content_text(last_user_msg.content) - attrs['input.value'] = truncate_content(content, max_length) + if langsmith_compat + last_user_msg = messages.reverse.find { |m| m.role.to_s == 'user' } + if last_user_msg + content = extract_content_text(last_user_msg.content) + attrs['input.value'] = truncate_content(content, max_length) + end end attrs end - def build_completion_attributes(message, max_length:) + def build_completion_attributes(message, max_length:, langsmith_compat: false) attrs = {} attrs['gen_ai.completion.0.role'] = message.role.to_s content = extract_content_text(message.content) truncated = truncate_content(content, max_length) attrs['gen_ai.completion.0.content'] = truncated # Set output.value for LangSmith Output panel - attrs['output.value'] = truncated + attrs['output.value'] = truncated if langsmith_compat attrs end - def build_request_attributes(model:, provider:, session_id:, temperature: nil, metadata: nil) + def build_request_attributes(model:, provider:, session_id:, temperature: nil, metadata: nil, + langsmith_compat: false, metadata_prefix: 'metadata') attrs = { - 'langsmith.span.kind' => 'LLM', 'gen_ai.system' => provider.to_s, 'gen_ai.operation.name' => 'chat', 'gen_ai.request.model' => model.id, 'gen_ai.conversation.id' => session_id } + attrs['langsmith.span.kind'] = 'LLM' if langsmith_compat attrs['gen_ai.request.temperature'] = temperature if temperature - build_metadata_attributes(attrs, metadata) if metadata + build_metadata_attributes(attrs, metadata, prefix: metadata_prefix) if metadata attrs end - def build_metadata_attributes(attrs, metadata, prefix: RubyLLM.config.tracing_metadata_prefix) + def build_metadata_attributes(attrs, metadata, prefix: 'metadata') metadata.each do |key, value| next if value.nil? @@ -188,33 +191,32 @@ def build_response_attributes(response) attrs end - def build_tool_attributes(tool_call:, session_id:) - { - 'langsmith.span.kind' => 'TOOL', + def build_tool_attributes(tool_call:, session_id:, langsmith_compat: false) + attrs = { 'gen_ai.operation.name' => 'tool', 'gen_ai.tool.name' => tool_call.name.to_s, 'gen_ai.tool.call.id' => tool_call.id, 'gen_ai.conversation.id' => session_id } + attrs['langsmith.span.kind'] = 'TOOL' if langsmith_compat + attrs end - def build_tool_input_attributes(tool_call:, max_length:) + def build_tool_input_attributes(tool_call:, max_length:, langsmith_compat: false) args = tool_call.arguments - input = args.is_a?(String) ? args : args.to_json + input = args.is_a?(String) ? args : JSON.generate(args) truncated = truncate_content(input, max_length) - { - 'input.value' => truncated, # LangSmith Input panel - 'gen_ai.prompt' => truncated # GenAI convention fallback - } + attrs = { 'gen_ai.tool.input' => truncated } + attrs['input.value'] = truncated if langsmith_compat + attrs end - def build_tool_output_attributes(result:, max_length:) + def build_tool_output_attributes(result:, max_length:, langsmith_compat: false) output = result.is_a?(String) ? result : result.to_s truncated = truncate_content(output, max_length) - { - 'output.value' => truncated, # LangSmith Output panel - 'gen_ai.completion' => truncated # GenAI convention fallback - } + attrs = { 'gen_ai.tool.output' => truncated } + attrs['output.value'] = truncated if langsmith_compat + attrs end end end diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index 3608cc208..73a0fdc5e 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -24,6 +24,27 @@ expect(config.tracing_metadata_prefix).to eq 'metadata' end + it 'has tracing_langsmith_compat defaulting to false' do + config = RubyLLM::Configuration.new + expect(config.tracing_langsmith_compat).to be false + end + + it 'auto-sets tracing_metadata_prefix when enabling langsmith_compat' do + config = RubyLLM::Configuration.new + expect(config.tracing_metadata_prefix).to eq 'metadata' + + config.tracing_langsmith_compat = true + expect(config.tracing_metadata_prefix).to eq 'langsmith.metadata' + end + + it 'does not override custom tracing_metadata_prefix when enabling langsmith_compat' do + config = RubyLLM::Configuration.new + config.tracing_metadata_prefix = 'app.custom' + + config.tracing_langsmith_compat = true + expect(config.tracing_metadata_prefix).to eq 'app.custom' + end + it 'allows configuration via block' do RubyLLM.configure do |config| config.tracing_enabled = true @@ -178,6 +199,20 @@ expect(attrs['gen_ai.tool.name']).to eq 'get_weather' expect(attrs['gen_ai.tool.call.id']).to eq 'call_123' expect(attrs['gen_ai.conversation.id']).to eq 'session-abc' + expect(attrs).not_to have_key('langsmith.span.kind') + end + + it 'includes langsmith.span.kind when langsmith_compat is true' do + tool_call = instance_double(RubyLLM::ToolCall, name: 'get_weather', id: 'call_123', + arguments: { location: 'NYC' }) + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_tool_attributes( + tool_call: tool_call, + session_id: 'session-abc', + langsmith_compat: true + ) + + expect(attrs['langsmith.span.kind']).to eq 'TOOL' end end @@ -190,7 +225,21 @@ max_length: 50 ) - expect(attrs['input.value']).to include('[truncated]') + expect(attrs['gen_ai.tool.input']).to include('[truncated]') + expect(attrs).not_to have_key('input.value') + end + + it 'includes input.value when langsmith_compat is true' do + tool_call = instance_double(RubyLLM::ToolCall, arguments: { query: 'test' }) + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_tool_input_attributes( + tool_call: tool_call, + max_length: 100, + langsmith_compat: true + ) + + expect(attrs['gen_ai.tool.input']).to be_a(String) + expect(attrs['input.value']).to eq(attrs['gen_ai.tool.input']) end end @@ -203,7 +252,21 @@ max_length: 50 ) - expect(attrs['output.value']).to include('[truncated]') + expect(attrs['gen_ai.tool.output']).to include('[truncated]') + expect(attrs).not_to have_key('output.value') + end + + it 'includes output.value when langsmith_compat is true' do + result = 'test output' + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_tool_output_attributes( + result: result, + max_length: 100, + langsmith_compat: true + ) + + expect(attrs['gen_ai.tool.output']).to eq('test output') + expect(attrs['output.value']).to eq('test output') end end @@ -282,6 +345,76 @@ expect(attrs['gen_ai.prompt.0.content']).to eq 'Describe this image' end + + it 'does not include input.value by default' do + messages = [RubyLLM::Message.new(role: :user, content: 'Hello')] + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 1000) + + expect(attrs).not_to have_key('input.value') + end + + it 'includes input.value when langsmith_compat is true' do + messages = [RubyLLM::Message.new(role: :user, content: 'Hello')] + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes( + messages, + max_length: 1000, + langsmith_compat: true + ) + + expect(attrs['input.value']).to eq 'Hello' + end + end + + describe '.build_completion_attributes' do + it 'does not include output.value by default' do + message = RubyLLM::Message.new(role: :assistant, content: 'Hi there!') + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_completion_attributes(message, max_length: 1000) + + expect(attrs['gen_ai.completion.0.content']).to eq 'Hi there!' + expect(attrs).not_to have_key('output.value') + end + + it 'includes output.value when langsmith_compat is true' do + message = RubyLLM::Message.new(role: :assistant, content: 'Hi there!') + + attrs = RubyLLM::Instrumentation::SpanBuilder.build_completion_attributes( + message, + max_length: 1000, + langsmith_compat: true + ) + + expect(attrs['output.value']).to eq 'Hi there!' + end + end + + describe '.build_request_attributes' do + let(:model) { instance_double(RubyLLM::Model, id: 'gpt-4') } + + it 'does not include langsmith.span.kind by default' do + attrs = RubyLLM::Instrumentation::SpanBuilder.build_request_attributes( + model: model, + provider: :openai, + session_id: 'session-123' + ) + + expect(attrs['gen_ai.system']).to eq 'openai' + expect(attrs['gen_ai.request.model']).to eq 'gpt-4' + expect(attrs).not_to have_key('langsmith.span.kind') + end + + it 'includes langsmith.span.kind when langsmith_compat is true' do + attrs = RubyLLM::Instrumentation::SpanBuilder.build_request_attributes( + model: model, + provider: :openai, + session_id: 'session-123', + langsmith_compat: true + ) + + expect(attrs['langsmith.span.kind']).to eq 'LLM' + end end describe '.build_metadata_attributes' do From d87a4a5ed0206b42141074da4dadbae3c8ef2c09 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:54:17 -0800 Subject: [PATCH 07/19] Namespace AR session_id to avoid collisions across models Use "#{self.class.name}:#{id}" format (e.g., "ChatSession:123") instead of just the ID. This prevents session_id collisions when multiple models use acts_as_chat, ensuring traces are correctly grouped per-model in observability backends. --- lib/ruby_llm/active_record/acts_as_legacy.rb | 7 ++++--- lib/ruby_llm/active_record/chat_methods.rb | 2 +- spec/ruby_llm/active_record/acts_as_model_spec.rb | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/lib/ruby_llm/active_record/acts_as_legacy.rb b/lib/ruby_llm/active_record/acts_as_legacy.rb index 512b2bdfe..f9490d64a 100644 --- a/lib/ruby_llm/active_record/acts_as_legacy.rb +++ b/lib/ruby_llm/active_record/acts_as_legacy.rb @@ -88,11 +88,12 @@ module ChatLegacyMethods def to_llm(context: nil) # model_id is a string that RubyLLM can resolve - # session_id uses the AR record ID (as string) for tracing session grouping + # session_id namespaced with class name for uniqueness across models + session = "#{self.class.name}:#{id}" @chat ||= if context - context.chat(model: model_id, session_id: id.to_s) + context.chat(model: model_id, session_id: session) else - RubyLLM.chat(model: model_id, session_id: id.to_s) + RubyLLM.chat(model: model_id, session_id: session) end @chat.reset_messages! diff --git a/lib/ruby_llm/active_record/chat_methods.rb b/lib/ruby_llm/active_record/chat_methods.rb index c8f97f5af..fa9bea11d 100644 --- a/lib/ruby_llm/active_record/chat_methods.rb +++ b/lib/ruby_llm/active_record/chat_methods.rb @@ -80,7 +80,7 @@ def to_llm @chat ||= (context || RubyLLM).chat( model: model_record.model_id, provider: model_record.provider.to_sym, - session_id: id.to_s # Use AR record ID for tracing session grouping + session_id: "#{self.class.name}:#{id}" # Namespaced for uniqueness across models ) @chat.reset_messages! diff --git a/spec/ruby_llm/active_record/acts_as_model_spec.rb b/spec/ruby_llm/active_record/acts_as_model_spec.rb index 14a62ae2d..e72c6731d 100644 --- a/spec/ruby_llm/active_record/acts_as_model_spec.rb +++ b/spec/ruby_llm/active_record/acts_as_model_spec.rb @@ -256,7 +256,7 @@ def messages expect(RubyLLM).to receive(:chat).with( # rubocop:disable RSpec/MessageSpies,RSpec/StubbedMock model: 'test-gpt', provider: :openai, - session_id: chat.id.to_s + session_id: "#{chat.class.name}:#{chat.id}" ).and_return( instance_double(RubyLLM::Chat, reset_messages!: nil, add_message: nil, instance_variable_get: {}, on_new_message: nil, on_end_message: nil, @@ -274,7 +274,7 @@ def messages expect(RubyLLM).to receive(:chat).with( # rubocop:disable RSpec/MessageSpies,RSpec/StubbedMock model: 'test-claude', provider: :anthropic, - session_id: chat.id.to_s + session_id: "#{chat.class.name}:#{chat.id}" ).and_return( instance_double(RubyLLM::Chat, reset_messages!: nil, add_message: nil, instance_variable_get: {}, on_new_message: nil, on_end_message: nil, From b83ea812e677e1d47267c17ea0cade2de063defe Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 11:55:07 -0800 Subject: [PATCH 08/19] Fix docs to reflect actual span attributes Update What Gets Traced section to show gen_ai.tool.input/output (always emitted) vs input.value/output.value (LangSmith-only). Add LangSmith-specific attribute tables to both Chat and Tool sections. --- docs/_advanced/observability.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md index 0edc4fbdd..c59116b6e 100644 --- a/docs/_advanced/observability.md +++ b/docs/_advanced/observability.md @@ -180,6 +180,7 @@ Each call to `chat.ask()` creates a `ruby_llm.chat` span with: | Attribute | Description | |-----------|-------------| | `gen_ai.system` | Provider name (openai, anthropic, etc.) | +| `gen_ai.operation.name` | Set to `chat` | | `gen_ai.request.model` | Requested model ID | | `gen_ai.request.temperature` | Temperature setting (if specified) | | `gen_ai.response.model` | Actual model used | @@ -187,6 +188,14 @@ Each call to `chat.ask()` creates a `ruby_llm.chat` span with: | `gen_ai.usage.output_tokens` | Output token count | | `gen_ai.conversation.id` | Session ID for grouping conversations | +When `tracing_langsmith_compat = true`, additional attributes are added: + +| Attribute | Description | +|-----------|-------------| +| `langsmith.span.kind` | Set to `LLM` | +| `input.value` | Last user message (for LangSmith Input panel) | +| `output.value` | Assistant response (for LangSmith Output panel) | + ### Tool Calls When tools are invoked, child `ruby_llm.tool` spans are created with: @@ -195,8 +204,17 @@ When tools are invoked, child `ruby_llm.tool` spans are created with: |-----------|-------------| | `gen_ai.tool.name` | Name of the tool | | `gen_ai.tool.call.id` | Unique call identifier | -| `input.value` | Tool arguments (if content logging enabled) | -| `output.value` | Tool result (if content logging enabled) | +| `gen_ai.conversation.id` | Session ID for grouping | +| `gen_ai.tool.input` | Tool arguments (if content logging enabled) | +| `gen_ai.tool.output` | Tool result (if content logging enabled) | + +When `tracing_langsmith_compat = true`, additional attributes are added: + +| Attribute | Description | +|-----------|-------------| +| `langsmith.span.kind` | Set to `TOOL` | +| `input.value` | Tool arguments (for LangSmith Input panel) | +| `output.value` | Tool result (for LangSmith Output panel) | ### Content Logging From d59a70618c3df7fc3143797ab598726ec61fe171 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 12:06:36 -0800 Subject: [PATCH 09/19] Address review feedback: safer tracing behavior - Remove tracer memoization to handle OTel reconfiguration - Warn and disable (not crash) when OTel gems missing (follows OTel's 'safe by default' pattern) - Revert metadata prefix when disabling langsmith_compat - JSON encode Hash/Array metadata values instead of .to_s garbage --- lib/ruby_llm/configuration.rb | 11 +++-- lib/ruby_llm/instrumentation.rb | 35 +++++++++------- spec/ruby_llm/instrumentation_spec.rb | 60 ++++++++++++++++++++++++--- 3 files changed, 83 insertions(+), 23 deletions(-) diff --git a/lib/ruby_llm/configuration.rb b/lib/ruby_llm/configuration.rb index 475a09a18..52ba16d5f 100644 --- a/lib/ruby_llm/configuration.rb +++ b/lib/ruby_llm/configuration.rb @@ -85,9 +85,14 @@ def initialize def tracing_langsmith_compat=(value) @tracing_langsmith_compat = value - # Auto-set metadata prefix for LangSmith when enabling compat mode, - # but only if the user hasn't customized it - @tracing_metadata_prefix = 'langsmith.metadata' if value && @tracing_metadata_prefix == 'metadata' + if value + # Auto-set metadata prefix for LangSmith when enabling compat mode, + # but only if the user hasn't customized it + @tracing_metadata_prefix = 'langsmith.metadata' if @tracing_metadata_prefix == 'metadata' + elsif @tracing_metadata_prefix == 'langsmith.metadata' + # Revert to default when disabling compat mode (if still using langsmith prefix) + @tracing_metadata_prefix = 'metadata' + end end def instance_variables diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index afabbfd6e..977db2ea1 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -16,16 +16,9 @@ def enabled?(config = RubyLLM.config) return false unless config.tracing_enabled unless otel_available? - raise RubyLLM::ConfigurationError, <<~MSG.strip - Tracing is enabled but OpenTelemetry is not available. - Please add the following gems to your Gemfile: - - gem 'opentelemetry-sdk' - gem 'opentelemetry-exporter-otlp' - - Then run `bundle install` and configure OpenTelemetry in an initializer. - See https://rubyllm.com/advanced/observability for setup instructions. - MSG + warn_otel_missing unless @otel_warning_issued + @otel_warning_issued = true + return false end true @@ -34,14 +27,12 @@ def enabled?(config = RubyLLM.config) def tracer(config = RubyLLM.config) return NullTracer.instance unless enabled?(config) - @tracer ||= OpenTelemetry.tracer_provider.tracer( - 'ruby_llm', - RubyLLM::VERSION - ) + # Don't memoize - tracer_provider can be reconfigured after initial load + OpenTelemetry.tracer_provider.tracer('ruby_llm', RubyLLM::VERSION) end def reset! - @tracer = nil + @otel_warning_issued = false end private @@ -51,6 +42,18 @@ def otel_available? !!OpenTelemetry.tracer_provider end + + def warn_otel_missing + RubyLLM.logger.warn <<~MSG.strip + [RubyLLM] Tracing is enabled but OpenTelemetry is not available. + Tracing will be disabled. To enable, add to your Gemfile: + + gem 'opentelemetry-sdk' + gem 'opentelemetry-exporter-otlp' + + See https://rubyllm.com/advanced/observability for setup instructions. + MSG + end end # No-op tracer used when tracing is disabled or OpenTelemetry is not available @@ -178,6 +181,8 @@ def otel_safe_value(value) case value when String, Integer, Float, TrueClass, FalseClass value + when Hash, Array + JSON.generate(value) else value.to_s end diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index 73a0fdc5e..ce8e73ec3 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -45,6 +45,24 @@ expect(config.tracing_metadata_prefix).to eq 'app.custom' end + it 'reverts tracing_metadata_prefix when disabling langsmith_compat' do + config = RubyLLM::Configuration.new + config.tracing_langsmith_compat = true + expect(config.tracing_metadata_prefix).to eq 'langsmith.metadata' + + config.tracing_langsmith_compat = false + expect(config.tracing_metadata_prefix).to eq 'metadata' + end + + it 'does not revert custom prefix when disabling langsmith_compat' do + config = RubyLLM::Configuration.new + config.tracing_langsmith_compat = true + config.tracing_metadata_prefix = 'app.custom' + + config.tracing_langsmith_compat = false + expect(config.tracing_metadata_prefix).to eq 'app.custom' + end + it 'allows configuration via block' do RubyLLM.configure do |config| config.tracing_enabled = true @@ -73,14 +91,24 @@ expect(described_class.enabled?).to be true end - it 'raises an error when tracing_enabled is true but OpenTelemetry is not available' do + it 'returns false and warns when tracing_enabled is true but OpenTelemetry is not available' do RubyLLM.configure { |c| c.tracing_enabled = true } + described_class.reset! allow(described_class).to receive(:otel_available?).and_return(false) - expect { described_class.enabled? }.to raise_error( - RubyLLM::ConfigurationError, - /OpenTelemetry is not available/ - ) + expect(RubyLLM.logger).to receive(:warn).with(/OpenTelemetry is not available/) + expect(described_class.enabled?).to be false + end + + it 'only warns once per reset cycle' do + RubyLLM.configure { |c| c.tracing_enabled = true } + described_class.reset! + allow(described_class).to receive(:otel_available?).and_return(false) + + expect(RubyLLM.logger).to receive(:warn).once + described_class.enabled? + described_class.enabled? + described_class.enabled? end end @@ -468,6 +496,28 @@ def complex_obj.to_s = 'complex_value' expect(attrs['test.data']).to eq 'complex_value' end + + it 'JSON encodes Hash values' do + attrs = {} + RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( + attrs, + { config: { nested: 'value', count: 42 } }, + prefix: 'test' + ) + + expect(attrs['test.config']).to eq '{"nested":"value","count":42}' + end + + it 'JSON encodes Array values' do + attrs = {} + RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( + attrs, + { tags: %w[foo bar baz] }, + prefix: 'test' + ) + + expect(attrs['test.tags']).to eq '["foo","bar","baz"]' + end end end From 3667cb45e7205f90c4fdeb1d3634fa1782dcd6fe Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 12:08:13 -0800 Subject: [PATCH 10/19] Simplify metadata handling: let OTel SDK handle type coercion Remove otel_safe_value method - just pass values through and let the OpenTelemetry SDK handle type conversion. No need to reinvent the wheel. --- lib/ruby_llm/instrumentation.rb | 16 ++---------- spec/ruby_llm/instrumentation_spec.rb | 36 ++++++--------------------- 2 files changed, 10 insertions(+), 42 deletions(-) diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index 977db2ea1..61f219fb8 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -171,20 +171,8 @@ def build_metadata_attributes(attrs, metadata, prefix: 'metadata') metadata.each do |key, value| next if value.nil? - # Preserve native types that OTel supports (string, int, float, bool) - # Only stringify complex objects - attrs["#{prefix}.#{key}"] = otel_safe_value(value) - end - end - - def otel_safe_value(value) - case value - when String, Integer, Float, TrueClass, FalseClass - value - when Hash, Array - JSON.generate(value) - else - value.to_s + # Let OTel SDK handle type coercion for supported types + attrs["#{prefix}.#{key}"] = value end end diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index ce8e73ec3..efcd497a7 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -446,7 +446,7 @@ end describe '.build_metadata_attributes' do - it 'builds attributes with the given prefix preserving native types' do + it 'builds attributes with the given prefix' do attrs = {} RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( attrs, @@ -483,40 +483,20 @@ expect(attrs).not_to have_key('test.empty') end - it 'stringifies complex objects' do + it 'passes values through for OTel SDK to handle' do attrs = {} - complex_obj = Object.new - def complex_obj.to_s = 'complex_value' + hash_value = { nested: 'value' } + array_value = %w[foo bar] RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( attrs, - { data: complex_obj }, + { config: hash_value, tags: array_value }, prefix: 'test' ) - expect(attrs['test.data']).to eq 'complex_value' - end - - it 'JSON encodes Hash values' do - attrs = {} - RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( - attrs, - { config: { nested: 'value', count: 42 } }, - prefix: 'test' - ) - - expect(attrs['test.config']).to eq '{"nested":"value","count":42}' - end - - it 'JSON encodes Array values' do - attrs = {} - RubyLLM::Instrumentation::SpanBuilder.build_metadata_attributes( - attrs, - { tags: %w[foo bar baz] }, - prefix: 'test' - ) - - expect(attrs['test.tags']).to eq '["foo","bar","baz"]' + # Values passed through as-is; OTel SDK handles type coercion + expect(attrs['test.config']).to eq hash_value + expect(attrs['test.tags']).to eq array_value end end end From 62ee78bd0f41c1c2bed6d7ed07c8229ea08f2ba0 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 13:37:20 -0800 Subject: [PATCH 11/19] Fix rubocop offenses and test failures - Use attr_reader for tracing_langsmith_compat (custom setter defined) - Consolidate build_request_attributes params into config hash - Use allow/have_received pattern for logger spy tests - Fix Model::Info double in tests --- lib/ruby_llm/chat.rb | 10 ++++++---- lib/ruby_llm/configuration.rb | 5 +++-- lib/ruby_llm/instrumentation.rb | 9 ++++----- spec/ruby_llm/instrumentation_spec.rb | 22 ++++++++++++++++++---- 4 files changed, 31 insertions(+), 15 deletions(-) diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index 9d27e1b16..ba768b4d9 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -168,10 +168,12 @@ def complete_with_span(span, &) model: @model, provider: @provider.slug, session_id: @session_id, - temperature: @temperature, - metadata: @metadata, - langsmith_compat: langsmith, - metadata_prefix: @config.tracing_metadata_prefix + config: { + temperature: @temperature, + metadata: @metadata, + langsmith_compat: langsmith, + metadata_prefix: @config.tracing_metadata_prefix + } ) ) diff --git a/lib/ruby_llm/configuration.rb b/lib/ruby_llm/configuration.rb index 52ba16d5f..482cb155d 100644 --- a/lib/ruby_llm/configuration.rb +++ b/lib/ruby_llm/configuration.rb @@ -51,8 +51,9 @@ class Configuration :tracing_enabled, :tracing_log_content, :tracing_max_content_length, - :tracing_metadata_prefix, - :tracing_langsmith_compat + :tracing_metadata_prefix + + attr_reader :tracing_langsmith_compat def initialize @request_timeout = 300 diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index 61f219fb8..fd21be501 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -153,17 +153,16 @@ def build_completion_attributes(message, max_length:, langsmith_compat: false) attrs end - def build_request_attributes(model:, provider:, session_id:, temperature: nil, metadata: nil, - langsmith_compat: false, metadata_prefix: 'metadata') + def build_request_attributes(model:, provider:, session_id:, config: {}) attrs = { 'gen_ai.system' => provider.to_s, 'gen_ai.operation.name' => 'chat', 'gen_ai.request.model' => model.id, 'gen_ai.conversation.id' => session_id } - attrs['langsmith.span.kind'] = 'LLM' if langsmith_compat - attrs['gen_ai.request.temperature'] = temperature if temperature - build_metadata_attributes(attrs, metadata, prefix: metadata_prefix) if metadata + attrs['langsmith.span.kind'] = 'LLM' if config[:langsmith_compat] + attrs['gen_ai.request.temperature'] = config[:temperature] if config[:temperature] + build_metadata_attributes(attrs, config[:metadata], prefix: config[:metadata_prefix]) if config[:metadata] attrs end diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index efcd497a7..4df498193 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -95,20 +95,23 @@ RubyLLM.configure { |c| c.tracing_enabled = true } described_class.reset! allow(described_class).to receive(:otel_available?).and_return(false) + allow(RubyLLM.logger).to receive(:warn) - expect(RubyLLM.logger).to receive(:warn).with(/OpenTelemetry is not available/) expect(described_class.enabled?).to be false + expect(RubyLLM.logger).to have_received(:warn).with(/OpenTelemetry is not available/) end it 'only warns once per reset cycle' do RubyLLM.configure { |c| c.tracing_enabled = true } described_class.reset! allow(described_class).to receive(:otel_available?).and_return(false) + allow(RubyLLM.logger).to receive(:warn) - expect(RubyLLM.logger).to receive(:warn).once described_class.enabled? described_class.enabled? described_class.enabled? + + expect(RubyLLM.logger).to have_received(:warn).once end end @@ -419,7 +422,7 @@ end describe '.build_request_attributes' do - let(:model) { instance_double(RubyLLM::Model, id: 'gpt-4') } + let(:model) { instance_double(RubyLLM::Model::Info, id: 'gpt-4') } it 'does not include langsmith.span.kind by default' do attrs = RubyLLM::Instrumentation::SpanBuilder.build_request_attributes( @@ -438,11 +441,22 @@ model: model, provider: :openai, session_id: 'session-123', - langsmith_compat: true + config: { langsmith_compat: true } ) expect(attrs['langsmith.span.kind']).to eq 'LLM' end + + it 'includes temperature when provided' do + attrs = RubyLLM::Instrumentation::SpanBuilder.build_request_attributes( + model: model, + provider: :openai, + session_id: 'session-123', + config: { temperature: 0.7 } + ) + + expect(attrs['gen_ai.request.temperature']).to eq 0.7 + end end describe '.build_metadata_attributes' do From 2791c857f32be05d443e8f56452173de7e72f3f1 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 14:13:38 -0800 Subject: [PATCH 12/19] Refactor metadata handling in build_request_attributes --- lib/ruby_llm/instrumentation.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index fd21be501..fe4dede7b 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -11,6 +11,7 @@ module SpanKind CLIENT = :client INTERNAL = :internal end + class << self def enabled?(config = RubyLLM.config) return false unless config.tracing_enabled @@ -162,7 +163,10 @@ def build_request_attributes(model:, provider:, session_id:, config: {}) } attrs['langsmith.span.kind'] = 'LLM' if config[:langsmith_compat] attrs['gen_ai.request.temperature'] = config[:temperature] if config[:temperature] - build_metadata_attributes(attrs, config[:metadata], prefix: config[:metadata_prefix]) if config[:metadata] + if config[:metadata]&.any? + build_metadata_attributes(attrs, config[:metadata], + prefix: config[:metadata_prefix]) + end attrs end From 9b270d96afe527c765387a3897ce5858328cbe9d Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 14:27:36 -0800 Subject: [PATCH 13/19] Add streaming instrumentation support - Unified complete() to always wrap in span (streaming + non-streaming) - Extracted add_request_span_attributes and add_response_span_attributes helpers - Added streaming documentation section - Added 4 streaming instrumentation tests --- docs/_advanced/observability.md | 23 +++++- lib/ruby_llm/chat.rb | 97 ++++++++++------------ spec/ruby_llm/instrumentation_spec.rb | 115 ++++++++++++++++++++++++++ 3 files changed, 179 insertions(+), 56 deletions(-) diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md index c59116b6e..8710a0809 100644 --- a/docs/_advanced/observability.md +++ b/docs/_advanced/observability.md @@ -25,6 +25,7 @@ After reading this guide, you will know: * How to enable OpenTelemetry tracing in RubyLLM * How to configure backends like LangSmith, DataDog, and Jaeger +* How streaming and non-streaming requests are traced * How session tracking groups multi-turn conversations * How to add custom metadata to traces * What attributes are captured in spans @@ -34,10 +35,10 @@ After reading this guide, you will know: | Feature | Status | |---------|--------| | Chat completions | ✅ Supported | +| Streaming | ✅ Supported | | Tool calls | ✅ Supported | | Session tracking | ✅ Supported | | Content logging (opt-in) | ✅ Supported | -| Streaming | ❌ Not yet supported | | Embeddings | ❌ Not yet supported | | Image generation | ❌ Not yet supported | | Transcription | ❌ Not yet supported | @@ -196,6 +197,26 @@ When `tracing_langsmith_compat = true`, additional attributes are added: | `input.value` | Last user message (for LangSmith Input panel) | | `output.value` | Assistant response (for LangSmith Output panel) | +### Streaming + +Streaming responses are traced identically to non-streaming responses. The span wraps the entire streaming operation: + +```ruby +chat.ask("Write a poem") do |chunk| + print chunk.content # Chunks stream in real-time +end +# Span completes here with full token counts +``` + +**How it works:** + +1. Span starts when `ask()` is called +2. Chunks stream to your block as they arrive +3. RubyLLM aggregates chunks internally +4. When streaming completes, token counts and final content are recorded on the span + +This follows the industry standard (LangSmith, Vercel AI SDK) where streaming operations get a single span representing the full request, not per-chunk spans. Tool calls during streaming create child spans just like non-streaming. + ### Tool Calls When tools are invoked, child `ruby_llm.tool` spans are created with: diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index ba768b4d9..580f07d06 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -133,9 +133,6 @@ def each(&) end def complete(&) - # Skip instrumentation for streaming (not supported yet) - return complete_without_instrumentation(&) if block_given? - Instrumentation.tracer(@config).in_span('ruby_llm.chat', kind: Instrumentation::SpanKind::CLIENT) do |span| complete_with_span(span, &) end @@ -143,7 +140,9 @@ def complete(&) private - def complete_without_instrumentation(&) + def complete_with_span(span, &) + add_request_span_attributes(span) + response = @provider.complete( messages, tools: @tools, @@ -155,69 +154,57 @@ def complete_without_instrumentation(&) &wrap_streaming_block(&) ) + add_response_span_attributes(span, response) finalize_response(response, &) + rescue StandardError => e + record_span_error(span, e) + raise end - def complete_with_span(span, &) + def add_request_span_attributes(span) + return unless span.recording? + langsmith = @config.tracing_langsmith_compat - # Set request attributes - if span.recording? - span.add_attributes( - Instrumentation::SpanBuilder.build_request_attributes( - model: @model, - provider: @provider.slug, - session_id: @session_id, - config: { - temperature: @temperature, - metadata: @metadata, - langsmith_compat: langsmith, - metadata_prefix: @config.tracing_metadata_prefix - } - ) + span.add_attributes( + Instrumentation::SpanBuilder.build_request_attributes( + model: @model, + provider: @provider.slug, + session_id: @session_id, + config: { + temperature: @temperature, + metadata: @metadata, + langsmith_compat: langsmith, + metadata_prefix: @config.tracing_metadata_prefix + } ) + ) - # Log message content if enabled - if @config.tracing_log_content - span.add_attributes( - Instrumentation::SpanBuilder.build_message_attributes( - messages, - max_length: @config.tracing_max_content_length, - langsmith_compat: langsmith - ) - ) - end - end + return unless @config.tracing_log_content - response = @provider.complete( - messages, - tools: @tools, - temperature: @temperature, - model: @model, - params: @params, - headers: @headers, - schema: @schema + span.add_attributes( + Instrumentation::SpanBuilder.build_message_attributes( + messages, + max_length: @config.tracing_max_content_length, + langsmith_compat: langsmith + ) ) + end - # Add response attributes - if span.recording? - span.add_attributes(Instrumentation::SpanBuilder.build_response_attributes(response)) + def add_response_span_attributes(span, response) + return unless span.recording? - if @config.tracing_log_content - span.add_attributes( - Instrumentation::SpanBuilder.build_completion_attributes( - response, - max_length: @config.tracing_max_content_length, - langsmith_compat: langsmith - ) - ) - end - end + span.add_attributes(Instrumentation::SpanBuilder.build_response_attributes(response)) - finalize_response(response, &) - rescue StandardError => e - record_span_error(span, e) - raise + return unless @config.tracing_log_content + + span.add_attributes( + Instrumentation::SpanBuilder.build_completion_attributes( + response, + max_length: @config.tracing_max_content_length, + langsmith_compat: @config.tracing_langsmith_compat + ) + ) end def record_span_error(span, exception) diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index 4df498193..aa6d050cd 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -730,5 +730,120 @@ def self.name expect(chat_span.status.code).to eq(OpenTelemetry::Trace::Status::ERROR) expect(chat_span.events.any? { |e| e.name == 'exception' }).to be true end + + describe 'Streaming' do + it 'creates spans for streaming responses' do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + + # Mock streaming response - provider.complete returns a Message even for streaming + mock_response = RubyLLM::Message.new( + role: :assistant, + content: 'Hello there!', + input_tokens: 8, + output_tokens: 3, + model_id: 'gpt-4.1-nano' + ) + allow(chat.instance_variable_get(:@provider)).to receive(:complete).and_return(mock_response) + + chunks = [] + chat.ask('Hello') { |chunk| chunks << chunk } + + spans = exporter.finished_spans + chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } + + expect(chat_span).not_to be_nil + expect(chat_span.kind).to eq(:client) + expect(chat_span.attributes['gen_ai.system']).to eq('openai') + expect(chat_span.attributes['gen_ai.usage.input_tokens']).to eq(8) + expect(chat_span.attributes['gen_ai.usage.output_tokens']).to eq(3) + end + + it 'includes completion content in streaming spans when content logging enabled' do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + + mock_response = RubyLLM::Message.new( + role: :assistant, + content: 'Streamed response content', + input_tokens: 5, + output_tokens: 4, + model_id: 'gpt-4.1-nano' + ) + allow(chat.instance_variable_get(:@provider)).to receive(:complete).and_return(mock_response) + + chat.ask('Test') { |_chunk| nil } + + spans = exporter.finished_spans + chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } + + expect(chat_span.attributes['gen_ai.completion.0.content']).to eq('Streamed response content') + end + + it 'maintains session_id for streaming calls' do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + session_id = chat.session_id + + mock_response = RubyLLM::Message.new( + role: :assistant, + content: 'Response', + model_id: 'gpt-4.1-nano' + ) + allow(chat.instance_variable_get(:@provider)).to receive(:complete).and_return(mock_response) + + chat.ask('First') { |_| nil } + chat.ask('Second') { |_| nil } + + spans = exporter.finished_spans + chat_spans = spans.select { |s| s.name == 'ruby_llm.chat' } + + expect(chat_spans.count).to eq(2) + expect(chat_spans.all? { |s| s.attributes['gen_ai.conversation.id'] == session_id }).to be true + end + + it 'creates tool spans as children during streaming with tools' do + chat = RubyLLM.chat(model: 'gpt-4.1-nano', assume_model_exists: true, provider: :openai) + chat.with_tool(weather_tool) + + # First response triggers tool call + tool_call_response = RubyLLM::Message.new( + role: :assistant, + content: nil, + tool_calls: { + 'call_123' => RubyLLM::ToolCall.new( + id: 'call_123', + name: 'weather', + arguments: { latitude: '52.52', longitude: '13.41' } + ) + }, + model_id: 'gpt-4.1-nano' + ) + + # Second response is final answer + final_response = RubyLLM::Message.new( + role: :assistant, + content: 'The weather in Berlin is nice!', + model_id: 'gpt-4.1-nano' + ) + + call_count = 0 + allow(chat.instance_variable_get(:@provider)).to receive(:complete) do + call_count += 1 + call_count == 1 ? tool_call_response : final_response + end + + chat.ask('Weather in Berlin?') { |_| nil } + + spans = exporter.finished_spans + tool_span = spans.find { |s| s.name == 'ruby_llm.tool' } + chat_spans = spans.select { |s| s.name == 'ruby_llm.chat' } + + expect(chat_spans.count).to eq(2) # Initial call + follow-up after tool + expect(tool_span).not_to be_nil + expect(tool_span.attributes['gen_ai.tool.name']).to eq('weather') + + # Tool span should be child of first chat span + first_chat_span = chat_spans.min_by(&:start_timestamp) + expect(tool_span.parent_span_id).to eq(first_chat_span.span_id) + end + end end end From ec1fcff4eafd33309b48d55a452eb41328e29146 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 14:45:59 -0800 Subject: [PATCH 14/19] Align instrumentation attributes with OTEL GenAI semantic conventions - Rename gen_ai.system to gen_ai.provider.name - Rename gen_ai.tool.input to gen_ai.tool.call.arguments - Rename gen_ai.tool.output to gen_ai.tool.call.result - Change tool operation name from 'tool' to 'execute_tool' - Add links to OTEL GenAI spec in observability docs See: https://opentelemetry.io/docs/specs/semconv/gen-ai/ --- docs/_advanced/observability.md | 8 +++++--- lib/ruby_llm/instrumentation.rb | 8 ++++---- spec/ruby_llm/instrumentation_spec.rb | 18 +++++++++--------- 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md index 8710a0809..8fa224770 100644 --- a/docs/_advanced/observability.md +++ b/docs/_advanced/observability.md @@ -174,13 +174,15 @@ RubyLLM works with any OpenTelemetry-compatible backend. Configure the `opentele ## What Gets Traced +RubyLLM follows the [OpenTelemetry Semantic Conventions for GenAI](https://opentelemetry.io/docs/specs/semconv/gen-ai/) ([GitHub source](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai)). + ### Chat Completions Each call to `chat.ask()` creates a `ruby_llm.chat` span with: | Attribute | Description | |-----------|-------------| -| `gen_ai.system` | Provider name (openai, anthropic, etc.) | +| `gen_ai.provider.name` | Provider name (openai, anthropic, etc.) | | `gen_ai.operation.name` | Set to `chat` | | `gen_ai.request.model` | Requested model ID | | `gen_ai.request.temperature` | Temperature setting (if specified) | @@ -226,8 +228,8 @@ When tools are invoked, child `ruby_llm.tool` spans are created with: | `gen_ai.tool.name` | Name of the tool | | `gen_ai.tool.call.id` | Unique call identifier | | `gen_ai.conversation.id` | Session ID for grouping | -| `gen_ai.tool.input` | Tool arguments (if content logging enabled) | -| `gen_ai.tool.output` | Tool result (if content logging enabled) | +| `gen_ai.tool.call.arguments` | Tool arguments (if content logging enabled) | +| `gen_ai.tool.call.result` | Tool result (if content logging enabled) | When `tracing_langsmith_compat = true`, additional attributes are added: diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index fe4dede7b..b69629258 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -156,7 +156,7 @@ def build_completion_attributes(message, max_length:, langsmith_compat: false) def build_request_attributes(model:, provider:, session_id:, config: {}) attrs = { - 'gen_ai.system' => provider.to_s, + 'gen_ai.provider.name' => provider.to_s, 'gen_ai.operation.name' => 'chat', 'gen_ai.request.model' => model.id, 'gen_ai.conversation.id' => session_id @@ -189,7 +189,7 @@ def build_response_attributes(response) def build_tool_attributes(tool_call:, session_id:, langsmith_compat: false) attrs = { - 'gen_ai.operation.name' => 'tool', + 'gen_ai.operation.name' => 'execute_tool', 'gen_ai.tool.name' => tool_call.name.to_s, 'gen_ai.tool.call.id' => tool_call.id, 'gen_ai.conversation.id' => session_id @@ -202,7 +202,7 @@ def build_tool_input_attributes(tool_call:, max_length:, langsmith_compat: false args = tool_call.arguments input = args.is_a?(String) ? args : JSON.generate(args) truncated = truncate_content(input, max_length) - attrs = { 'gen_ai.tool.input' => truncated } + attrs = { 'gen_ai.tool.call.arguments' => truncated } attrs['input.value'] = truncated if langsmith_compat attrs end @@ -210,7 +210,7 @@ def build_tool_input_attributes(tool_call:, max_length:, langsmith_compat: false def build_tool_output_attributes(result:, max_length:, langsmith_compat: false) output = result.is_a?(String) ? result : result.to_s truncated = truncate_content(output, max_length) - attrs = { 'gen_ai.tool.output' => truncated } + attrs = { 'gen_ai.tool.call.result' => truncated } attrs['output.value'] = truncated if langsmith_compat attrs end diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index aa6d050cd..932285130 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -226,7 +226,7 @@ session_id: 'session-abc' ) - expect(attrs['gen_ai.operation.name']).to eq 'tool' + expect(attrs['gen_ai.operation.name']).to eq 'execute_tool' expect(attrs['gen_ai.tool.name']).to eq 'get_weather' expect(attrs['gen_ai.tool.call.id']).to eq 'call_123' expect(attrs['gen_ai.conversation.id']).to eq 'session-abc' @@ -256,7 +256,7 @@ max_length: 50 ) - expect(attrs['gen_ai.tool.input']).to include('[truncated]') + expect(attrs['gen_ai.tool.call.arguments']).to include('[truncated]') expect(attrs).not_to have_key('input.value') end @@ -269,8 +269,8 @@ langsmith_compat: true ) - expect(attrs['gen_ai.tool.input']).to be_a(String) - expect(attrs['input.value']).to eq(attrs['gen_ai.tool.input']) + expect(attrs['gen_ai.tool.call.arguments']).to be_a(String) + expect(attrs['input.value']).to eq(attrs['gen_ai.tool.call.arguments']) end end @@ -283,7 +283,7 @@ max_length: 50 ) - expect(attrs['gen_ai.tool.output']).to include('[truncated]') + expect(attrs['gen_ai.tool.call.result']).to include('[truncated]') expect(attrs).not_to have_key('output.value') end @@ -296,7 +296,7 @@ langsmith_compat: true ) - expect(attrs['gen_ai.tool.output']).to eq('test output') + expect(attrs['gen_ai.tool.call.result']).to eq('test output') expect(attrs['output.value']).to eq('test output') end end @@ -431,7 +431,7 @@ session_id: 'session-123' ) - expect(attrs['gen_ai.system']).to eq 'openai' + expect(attrs['gen_ai.provider.name']).to eq 'openai' expect(attrs['gen_ai.request.model']).to eq 'gpt-4' expect(attrs).not_to have_key('langsmith.span.kind') end @@ -663,7 +663,7 @@ def self.name chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } expect(chat_span).not_to be_nil expect(chat_span.kind).to eq(:client) - expect(chat_span.attributes['gen_ai.system']).to eq('openai') + expect(chat_span.attributes['gen_ai.provider.name']).to eq('openai') expect(chat_span.attributes['gen_ai.operation.name']).to eq('chat') expect(chat_span.attributes['gen_ai.conversation.id']).to be_a(String) expect(chat_span.attributes['gen_ai.prompt.0.role']).to eq('user') @@ -753,7 +753,7 @@ def self.name expect(chat_span).not_to be_nil expect(chat_span.kind).to eq(:client) - expect(chat_span.attributes['gen_ai.system']).to eq('openai') + expect(chat_span.attributes['gen_ai.provider.name']).to eq('openai') expect(chat_span.attributes['gen_ai.usage.input_tokens']).to eq(8) expect(chat_span.attributes['gen_ai.usage.output_tokens']).to eq(3) end From fdf85935bb02a0fca9de6c16fcde568f5f6facf4 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 14:50:39 -0800 Subject: [PATCH 15/19] Add JSON requirement for instrumentation module - Introduced 'json' library to enhance metadata handling capabilities. --- lib/ruby_llm/instrumentation.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index b69629258..a119451b1 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -1,5 +1,6 @@ # frozen_string_literal: true +require 'json' require 'singleton' module RubyLLM From 58a00483ea950a9d27260cd089575f57afc3f02d Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 14:52:53 -0800 Subject: [PATCH 16/19] Enhance observability logging to comply with OTEL GenAI spec - Updated content logging to output prompts and completions as JSON arrays. - Introduced `gen_ai.input.messages` and `gen_ai.output.messages` attributes for structured message logging. - Refactored `build_message_attributes` and `build_completion_attributes` methods to generate OTEL-compliant message formats. - Updated tests to validate new JSON structure for input and output messages. --- docs/_advanced/observability.md | 16 ++++++--- lib/ruby_llm/instrumentation.rb | 46 +++++++++++++++++++----- spec/ruby_llm/instrumentation_spec.rb | 52 +++++++++++++++++---------- 3 files changed, 81 insertions(+), 33 deletions(-) diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md index 8fa224770..2fdbdc33e 100644 --- a/docs/_advanced/observability.md +++ b/docs/_advanced/observability.md @@ -241,14 +241,20 @@ When `tracing_langsmith_compat = true`, additional attributes are added: ### Content Logging -When `tracing_log_content = true`, prompts and completions are logged: +When `tracing_log_content = true`, prompts and completions are logged as JSON arrays following the OTEL GenAI spec: | Attribute | Description | |-----------|-------------| -| `gen_ai.prompt.0.role` | Role of first message (user, system, assistant) | -| `gen_ai.prompt.0.content` | Content of first message | -| `gen_ai.completion.0.role` | Role of response | -| `gen_ai.completion.0.content` | Response content | +| `gen_ai.input.messages` | JSON array of input messages with role and parts | +| `gen_ai.output.messages` | JSON array of output messages with role and parts | + +Example `gen_ai.input.messages`: +```json +[ + {"role": "system", "parts": [{"type": "text", "content": "You are helpful"}]}, + {"role": "user", "parts": [{"type": "text", "content": "Hello"}]} +] +``` --- diff --git a/lib/ruby_llm/instrumentation.rb b/lib/ruby_llm/instrumentation.rb index a119451b1..4ea490921 100644 --- a/lib/ruby_llm/instrumentation.rb +++ b/lib/ruby_llm/instrumentation.rb @@ -128,11 +128,13 @@ def describe_attachments(attachments) def build_message_attributes(messages, max_length:, langsmith_compat: false) attrs = {} - messages.each_with_index do |msg, idx| - attrs["gen_ai.prompt.#{idx}.role"] = msg.role.to_s - content = extract_content_text(msg.content) - attrs["gen_ai.prompt.#{idx}.content"] = truncate_content(content, max_length) + + # Build OTEL GenAI spec-compliant input messages + input_messages = messages.map do |msg| + build_message_object(msg) end + attrs['gen_ai.input.messages'] = truncate_content(JSON.generate(input_messages), max_length) + # Set input.value for LangSmith Input panel (last user message) if langsmith_compat last_user_msg = messages.reverse.find { |m| m.role.to_s == 'user' } @@ -146,15 +148,41 @@ def build_message_attributes(messages, max_length:, langsmith_compat: false) def build_completion_attributes(message, max_length:, langsmith_compat: false) attrs = {} - attrs['gen_ai.completion.0.role'] = message.role.to_s - content = extract_content_text(message.content) - truncated = truncate_content(content, max_length) - attrs['gen_ai.completion.0.content'] = truncated + + # Build OTEL GenAI spec-compliant output messages + output_messages = [build_message_object(message)] + attrs['gen_ai.output.messages'] = truncate_content(JSON.generate(output_messages), max_length) + # Set output.value for LangSmith Output panel - attrs['output.value'] = truncated if langsmith_compat + if langsmith_compat + content = extract_content_text(message.content) + attrs['output.value'] = truncate_content(content, max_length) + end attrs end + def build_message_object(message) + content = extract_content_text(message.content) + obj = { + role: message.role.to_s, + parts: [{ type: 'text', content: content }] + } + + # Add tool calls if present + if message.respond_to?(:tool_calls) && message.tool_calls&.any? + message.tool_calls.each_value do |tc| + obj[:parts] << { + type: 'tool_call', + id: tc.id, + name: tc.name, + arguments: tc.arguments + } + end + end + + obj + end + def build_request_attributes(model:, provider:, session_id:, config: {}) attrs = { 'gen_ai.provider.name' => provider.to_s, diff --git a/spec/ruby_llm/instrumentation_spec.rb b/spec/ruby_llm/instrumentation_spec.rb index 932285130..8cec220a4 100644 --- a/spec/ruby_llm/instrumentation_spec.rb +++ b/spec/ruby_llm/instrumentation_spec.rb @@ -346,41 +346,46 @@ end describe '.build_message_attributes' do - it 'builds indexed attributes for messages' do + it 'builds gen_ai.input.messages as JSON array' do messages = [ RubyLLM::Message.new(role: :system, content: 'You are helpful'), RubyLLM::Message.new(role: :user, content: 'Hello') ] - attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 1000) + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 10_000) - expect(attrs['gen_ai.prompt.0.role']).to eq 'system' - expect(attrs['gen_ai.prompt.0.content']).to eq 'You are helpful' - expect(attrs['gen_ai.prompt.1.role']).to eq 'user' - expect(attrs['gen_ai.prompt.1.content']).to eq 'Hello' + parsed = JSON.parse(attrs['gen_ai.input.messages']) + expect(parsed).to be_an(Array) + expect(parsed.length).to eq 2 + expect(parsed[0]['role']).to eq 'system' + expect(parsed[0]['parts'][0]['content']).to eq 'You are helpful' + expect(parsed[1]['role']).to eq 'user' + expect(parsed[1]['parts'][0]['content']).to eq 'Hello' end - it 'truncates long content' do + it 'truncates long JSON content' do messages = [RubyLLM::Message.new(role: :user, content: 'a' * 100)] attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 50) - expect(attrs['gen_ai.prompt.0.content']).to eq "#{'a' * 50}... [truncated]" + expect(attrs['gen_ai.input.messages']).to include('[truncated]') + expect(attrs['gen_ai.input.messages'].length).to be <= 70 # 50 + "... [truncated]" end it 'handles Content objects with attachments' do content = RubyLLM::Content.new('Describe this image', ['spec/fixtures/ruby.png']) messages = [RubyLLM::Message.new(role: :user, content: content)] - attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 1000) + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 10_000) - expect(attrs['gen_ai.prompt.0.content']).to eq 'Describe this image' + parsed = JSON.parse(attrs['gen_ai.input.messages']) + expect(parsed[0]['parts'][0]['content']).to eq 'Describe this image' end it 'does not include input.value by default' do messages = [RubyLLM::Message.new(role: :user, content: 'Hello')] - attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 1000) + attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes(messages, max_length: 10_000) expect(attrs).not_to have_key('input.value') end @@ -390,7 +395,7 @@ attrs = RubyLLM::Instrumentation::SpanBuilder.build_message_attributes( messages, - max_length: 1000, + max_length: 10_000, langsmith_compat: true ) @@ -399,12 +404,16 @@ end describe '.build_completion_attributes' do - it 'does not include output.value by default' do + it 'builds gen_ai.output.messages as JSON array' do message = RubyLLM::Message.new(role: :assistant, content: 'Hi there!') - attrs = RubyLLM::Instrumentation::SpanBuilder.build_completion_attributes(message, max_length: 1000) + attrs = RubyLLM::Instrumentation::SpanBuilder.build_completion_attributes(message, max_length: 10_000) - expect(attrs['gen_ai.completion.0.content']).to eq 'Hi there!' + parsed = JSON.parse(attrs['gen_ai.output.messages']) + expect(parsed).to be_an(Array) + expect(parsed.length).to eq 1 + expect(parsed[0]['role']).to eq 'assistant' + expect(parsed[0]['parts'][0]['content']).to eq 'Hi there!' expect(attrs).not_to have_key('output.value') end @@ -413,7 +422,7 @@ attrs = RubyLLM::Instrumentation::SpanBuilder.build_completion_attributes( message, - max_length: 1000, + max_length: 10_000, langsmith_compat: true ) @@ -666,8 +675,12 @@ def self.name expect(chat_span.attributes['gen_ai.provider.name']).to eq('openai') expect(chat_span.attributes['gen_ai.operation.name']).to eq('chat') expect(chat_span.attributes['gen_ai.conversation.id']).to be_a(String) - expect(chat_span.attributes['gen_ai.prompt.0.role']).to eq('user') - expect(chat_span.attributes['gen_ai.prompt.0.content']).to eq('Hello') + + # Check gen_ai.input.messages is valid JSON with correct structure + input_messages = JSON.parse(chat_span.attributes['gen_ai.input.messages']) + expect(input_messages).to be_an(Array) + expect(input_messages.last['role']).to eq('user') + expect(input_messages.last['parts'][0]['content']).to eq('Hello') end it 'creates tool spans as children when tools are used' do @@ -775,7 +788,8 @@ def self.name spans = exporter.finished_spans chat_span = spans.find { |s| s.name == 'ruby_llm.chat' } - expect(chat_span.attributes['gen_ai.completion.0.content']).to eq('Streamed response content') + output_messages = JSON.parse(chat_span.attributes['gen_ai.output.messages']) + expect(output_messages[0]['parts'][0]['content']).to eq('Streamed response content') end it 'maintains session_id for streaming calls' do From 2d156661b2213a8759026bc00d8bfe6ba2729147 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 15:15:08 -0800 Subject: [PATCH 17/19] Refactor span naming in chat and tool execution methods for improved observability - Updated span names in the complete and execute_tool methods to include specific identifiers for better traceability. - Enhanced logging clarity by dynamically generating span names based on model ID and tool call name. --- lib/ruby_llm/chat.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index 580f07d06..7aa7c83f3 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -133,7 +133,8 @@ def each(&) end def complete(&) - Instrumentation.tracer(@config).in_span('ruby_llm.chat', kind: Instrumentation::SpanKind::CLIENT) do |span| + span_name = "chat #{@model.id}" + Instrumentation.tracer(@config).in_span(span_name, kind: Instrumentation::SpanKind::CLIENT) do |span| complete_with_span(span, &) end end @@ -291,7 +292,8 @@ def handle_tool_calls(response, &) # rubocop:disable Metrics/PerceivedComplexity end def execute_tool(tool_call) - Instrumentation.tracer(@config).in_span('ruby_llm.tool', kind: Instrumentation::SpanKind::INTERNAL) do |span| + span_name = "execute_tool #{tool_call.name}" + Instrumentation.tracer(@config).in_span(span_name, kind: Instrumentation::SpanKind::INTERNAL) do |span| execute_tool_with_span(tool_call, span) end end From 439a15f741de602b2f588efd2a0c4fdcefa9bb3b Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 15:35:00 -0800 Subject: [PATCH 18/19] Update span naming in observability documentation for clarity - Revised span naming conventions in the documentation for `chat.ask()` and tool invocation to include specific identifiers, enhancing traceability and alignment with industry standards. --- docs/_advanced/observability.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/_advanced/observability.md b/docs/_advanced/observability.md index 2fdbdc33e..42119284b 100644 --- a/docs/_advanced/observability.md +++ b/docs/_advanced/observability.md @@ -178,7 +178,7 @@ RubyLLM follows the [OpenTelemetry Semantic Conventions for GenAI](https://opent ### Chat Completions -Each call to `chat.ask()` creates a `ruby_llm.chat` span with: +Each call to `chat.ask()` creates a `chat {model_id}` span (e.g., `chat gpt-4o`) with: | Attribute | Description | |-----------|-------------| @@ -221,7 +221,7 @@ This follows the industry standard (LangSmith, Vercel AI SDK) where streaming op ### Tool Calls -When tools are invoked, child `ruby_llm.tool` spans are created with: +When tools are invoked, child `execute_tool {tool_name}` spans (e.g., `execute_tool get_weather`) are created with: | Attribute | Description | |-----------|-------------| From 812e4082881cdf5bf965b2aeb22c95311a27da49 Mon Sep 17 00:00:00 2001 From: Philipp Comans Date: Tue, 6 Jan 2026 15:38:01 -0800 Subject: [PATCH 19/19] reorganize --- lib/ruby_llm/chat.rb | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/lib/ruby_llm/chat.rb b/lib/ruby_llm/chat.rb index 7aa7c83f3..8361c9e41 100644 --- a/lib/ruby_llm/chat.rb +++ b/lib/ruby_llm/chat.rb @@ -139,6 +139,20 @@ def complete(&) end end + def add_message(message_or_attributes) + message = message_or_attributes.is_a?(Message) ? message_or_attributes : Message.new(message_or_attributes) + messages << message + message + end + + def reset_messages! + @messages.clear + end + + def instance_variables + super - %i[@connection @config] + end + private def complete_with_span(span, &) @@ -238,24 +252,6 @@ def finalize_response(response, &) # rubocop:disable Metrics/PerceivedComplexity end end - public - - def add_message(message_or_attributes) - message = message_or_attributes.is_a?(Message) ? message_or_attributes : Message.new(message_or_attributes) - messages << message - message - end - - def reset_messages! - @messages.clear - end - - def instance_variables - super - %i[@connection @config] - end - - private - def wrap_streaming_block(&block) return nil unless block_given?