Auto-instrument LLM libraries with tracing

[delner]

This PR introduces automatic instrumentation for LLM libraries in Ruby applications. With minimal or zero code changes, all your LLM calls are automatically traced to Braintrust.

Features

Quick Start

Add to your Gemfile and your LLM calls are automatically traced:

gem "braintrust", require: "braintrust/setup"

export BRAINTRUST_API_KEY="your-api-key"
bundle install

That's it. View your traces at braintrust.dev.

Three Ways to Enable Auto-Instrumentation

Method	Code	Best For
Gemfile require	gem "braintrust", require: "braintrust/setup"	Rails apps, most Ruby apps
CLI wrapper	braintrust exec -- ruby app.rb	No code changes needed
Manual init	Braintrust.init in your code	Custom configuration

Supported LLM Libraries

Provider	Gem	Integration Name
OpenAI	openai (official)	:openai
OpenAI	ruby-openai	:ruby_openai
Anthropic	anthropic	:anthropic
Multiple	ruby_llm	:ruby_llm

Usage Examples

Example 1: Rails Application (Recommended)

# Gemfile
gem "braintrust", require: "braintrust/setup"
gem "openai"

# app/controllers/chat_controller.rb
class ChatController < ApplicationController
  def create
    client = OpenAI::Client.new
    # This call is automatically traced!
    response = client.chat.completions.create(
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: params[:message] }]
    )
    render json: { reply: response.choices[0].message.content }
  end
end

Example 2: CLI Wrapper (Zero Code Changes)

# Instrument any existing Ruby application
braintrust exec -- ruby my_app.rb
braintrust exec -- bundle exec rails server

# Filter which libraries to instrument
braintrust exec --only openai -- ruby app.rb
braintrust exec --except ruby_llm -- ruby app.rb

Example 3: Programmatic Control

require "braintrust"
require "openai"
require "anthropic"

# Full control over initialization
Braintrust.init(
  default_project: "my-project",
  auto_instrument: { except: [:anthropic] }
)

client = OpenAI::Client.new
client.chat.completions.create(...)  # Traced!

Example 4: Manual Instrumentation

require "braintrust"
require "openai"

Braintrust.init(auto_instrument: false)

# Instrument all clients of a type
Braintrust.instrument!(:openai)

# Or instrument a specific client instance
client = OpenAI::Client.new
Braintrust.instrument!(:openai, target: client)

Configuration Options

Environment Variables

Variable	Description
BRAINTRUST_API_KEY	Required. Your Braintrust API key
BRAINTRUST_AUTO_INSTRUMENT	Set to false to disable
BRAINTRUST_INSTRUMENT_ONLY	Comma-separated whitelist (e.g., openai,anthropic)
BRAINTRUST_INSTRUMENT_EXCEPT	Comma-separated blacklist
BRAINTRUST_DEFAULT_PROJECT	Default project for spans
BRAINTRUST_DEBUG	Set to true for debug logging

Braintrust.init Options

Braintrust.init(
  api_key: "...",                           # API key (default: env var)
  auto_instrument: true,                    # true, false, or Hash
  # auto_instrument: { only: [:openai] },     # Alternative: Whitelist
  # auto_instrument: { except: [:ruby_llm] }, # Alternative: Blacklist
  default_project: "my-project",            # Default project for spans
  blocking_login: false,                    # Wait for login (default: async), for short-lived processes.
)

---

Architecture Overview

Auto-Instrumentation Flow

┌─────────────────────────────────────────────────────────────────────────┐
│                         INSTALLATION METHODS                            │
├─────────────────────┬─────────────────────┬─────────────────────────────┤
│  require: "setup"   │   braintrust exec   │       Braintrust.init       │
│     (Gemfile)       │       (CLI)         │          (Code)             │
└─────────┬───────────┴──────────┬──────────┴──────────────┬──────────────┘
          │                      │                         │
          │            Sets RUBYOPT to                     │
          │            -rbraintrust/setup                  │
          │                      │                         │
          ▼                      ▼                         │
┌────────────────────────────────────────────────┐         │
│              braintrust/setup.rb               │         │
└───────────────────────┬────────────────────────┘         │
                        │                                  │
          ┌─────────────┴─────────────┐                    │
          │                           │                    │
          ▼                           ▼                    ▼
┌──────────────────────────┐   ┌─────────────────────────────────────────┐
│   Contrib::Setup.run!    │   │             Braintrust.init             │
│ (hooks future requires)  │   │   • Initializes tracing/login           │
└────────────┬─────────────┘   │   • Calls auto_instrument!              │
             │                 └────────────────────┬────────────────────┘
   ┌─────────┴─────────┐                            │
   ▼                   ▼                            │
┌─────────────────┐ ┌──────────────────────┐        │
│ Rails: Railtie  │ │ Other: Kernel#require│        │
│ after_initialize│ │ hook for future libs │        │
└────────┬────────┘ └──────────┬───────────┘        │
         │                     │                    │
         ▼                     ▼                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                    Braintrust.auto_instrument!                          │
│  • Queries Registry for available integrations                          │
│  • Applies only/except filters from env vars or options                 │
│  • Calls integration.instrument! for each matching integration          │
└─────────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                      Integration.instrument!                            │
│  • Checks if gem is available and compatible version                    │
│  • Delegates to Patcher classes for actual monkey-patching              │
│  • Thread-safe, idempotent patching                                     │
└─────────────────────────────────────────────────────────────────────────┘

Integration API

A new framework for defining LLM library integrations (PR #72):

┌──────────────────────────────────────────────────────────────────────┐
│                           Registry                                   │
│  • Singleton managing all integrations                               │
│  • Maps require paths → integrations for auto-detection              │
│  • Provides available() to find loaded libraries                     │
└───────────────────────────────┬──────────────────────────────────────┘
                                │ registers
                                ▼
┌──────────────────────────────────────────────────────────────────────┐
│                         Integration                                  │
│  • Module included in integration classes                            │
│  • Defines contract: integration_name, gem_names, loaded?, etc.      │
│  • Handles version compatibility checks                              │
│  • Delegates patching to Patcher classes                             │
└───────────────────────────────┬──────────────────────────────────────┘
                                │ uses
                                ▼
┌──────────────────────────────────────────────────────────────────────┐
│                           Patcher                                    │
│  • Base class for monkey-patching                                    │
│  • Thread-safe with mutex protection                                 │
│  • Idempotent (safe to call multiple times)                          │
│  • Subclasses implement perform_patch()                              │
└──────────────────────────────────────────────────────────────────────┘

Each integration (e.g., OpenAI) follows this structure:

lib/braintrust/contrib/openai/
├── integration.rb      # Integration metadata (gem names, versions, etc.)
├── patcher.rb          # Patching logic (ChatPatcher, ResponsesPatcher)
├── deprecated.rb       # Backward compatibility shims
└── instrumentation/    # Actual tracing instrumentation
    ├── common.rb       # Shared utilities
    ├── chat.rb         # Chat completions instrumentation
    └── responses.rb    # Responses API instrumentation

---

Constituent Pull Requests

PR	Description
#72	Integration framework API
#76	Migrated openai gem to new integration API
#78	Migrated ruby-openai gem to new integration API
#79	Migrated ruby_llm gem to new integration API
#80	Migrated anthropic gem to new integration API
#81	Added auto_instrument to Braintrust.init
#82	Added require "braintrust/setup" entry point
#83	Added braintrust exec CLI command
#84	Documentation updates and examples

[delner]

So I did a little benchmarking against my test Rails application (a simple Rails 8 app that pits LLMs in a Dad Joke contest of which Anthropic judges the winner)... it runs on Rails 8

Benchmark Results

Date: 2026-01-10 05:05:24
Iterations: 5 per scenario
Environment: Docker (ruby:3.2.9-slim)
Gems: rails 8.1.2, braintrust 0.0.12, openai 0.42.0, anthropic 1.16.3, ruby_llm 1.9.1

Cold Start

Scenario	Boot (ms)	Δ Boot	Memory (MB)	Δ Memory
No Braintrust	725.99	-	92.96	-
Gem loaded, no init	774.36	+48	94.15	+1.2
Braintrust.instrument! per integration	784.64	+59	94.89	+1.9
Braintrust.init	786.38	+60	96.97	+4.0
require 'braintrust/setup' from Gemfile	794.5	+69	97.05	+4.1
require 'braintrust/setup' from initializer	787.19	+61	97.04	+4.1
braintrust exec	933.36	+207	97.06	+4.1

Key Findings

• All in-process instrumentation approaches add ~45-60ms to boot time
• Memory overhead ranges from ~1MB (gem only) to ~4MB (full auto-setup)
• braintrust exec has significant overhead (~200ms) due to the CLI spawning a separate Ruby process
• The in-process approaches (init, setup, instrument!) are all roughly equivalent in boot time