Skip to content

Fix Bedrock non-streaming thinking_tokens always returning nil#714

Open
alexmamrenko wants to merge 1 commit intocrmne:mainfrom
alexmamrenko:fix/bedrock-thinking-tokens
Open

Fix Bedrock non-streaming thinking_tokens always returning nil#714
alexmamrenko wants to merge 1 commit intocrmne:mainfrom
alexmamrenko:fix/bedrock-thinking-tokens

Conversation

@alexmamrenko
Copy link
Copy Markdown

@alexmamrenko alexmamrenko commented Apr 1, 2026

What this does

The Bedrock Converse API returns thinking/reasoning token counts nested under usage.outputTokensDetails.reasoningTokens, not at the top-level usage.reasoningTokens.

The streaming parser already handles this correctly via extract_reasoning_tokens in bedrock/streaming.rb:

def extract_reasoning_tokens(metadata_usage, usage)
  metadata_usage['reasoningTokens'] || usage['reasoningTokens'] ||
    usage.dig('output_tokens_details', 'thinking_tokens')
end

But parse_completion_response in bedrock/chat.rb only checked the top-level key:

thinking_tokens: usage['reasoningTokens'],

This caused thinking_tokens to always be nil for non-streaming Bedrock requests (e.g. chat.ask).

The fix adds the nested key fallback to match the streaming parser's behavior:

thinking_tokens: usage['reasoningTokens'] || usage.dig('outputTokensDetails', 'reasoningTokens'),

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Quality check

  • I ran overcommit --install and all hooks pass
  • I tested my changes thoroughly
    • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
    • All tests pass: bundle exec rspec
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

  • I used AI tools to help write this code
  • I have reviewed and understand all generated code (required if above is checked)

API changes

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

@alexmamrenko
Copy link
Copy Markdown
Author

Failed rubocop is not related to my changes

@alexmamrenko alexmamrenko force-pushed the fix/bedrock-thinking-tokens branch from b8ff0db to e7ab477 Compare April 1, 2026 08:54
The Bedrock Converse API returns thinking token counts nested
under usage.outputTokensDetails.reasoningTokens, not at the
top-level usage.reasoningTokens. The streaming parser already
handles this fallback correctly, but parse_completion_response
only checked the top-level key.

Add fallback to nested path and tests covering both paths.
@alexmamrenko alexmamrenko force-pushed the fix/bedrock-thinking-tokens branch from e7ab477 to 9c943a3 Compare April 1, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant