feat(ai-proxy): add native Anthropic endpoint support #12936
Conversation
dosubot
bot
added
size:XL
There was a problem hiding this comment.
Pull request overview
This PR attempts to add native Anthropic API support to the ai-proxy plugin, providing first-class support for Anthropic's /v1/messages endpoint instead of relying on OpenAI-compatible adapters. The implementation includes a new Anthropic driver, base driver abstraction, tests, and documentation.
Changes:
- Added a native Anthropic driver implementation with request/response transformation methods
- Created a base AI driver class for shared functionality
- Modified test suite to validate Anthropic-specific functionality
- Added documentation for Anthropic provider usage
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 23 comments.
| File | Description |
|---|---|
| apisix/plugins/ai-drivers/anthropic.lua | New Anthropic driver with protocol translation methods, but missing critical required methods |
| apisix/plugins/ai-drivers/ai-driver-base.lua | New base driver class, but lacks essential functionality for AI drivers |
| t/plugin/ai-proxy-anthropic.t | Test suite rewritten to test native format, but doesn't test claimed protocol translation |
| docs/en/latest/plugins/ai-proxy-anthropic.md | Documentation describing features that are not actually implemented |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| local headers = { | ||
| ["Content-Type"] = "application/json", | ||
| ["x-api-key"] = conf.api_key, | ||
| ["anthropic-version"] = ANTHROPIC_VERSION, | ||
| } | ||
|
|
||
| return anthropic_body, headers |
There was a problem hiding this comment.
The transform_request method returns headers but doesn't handle the authentication scheme correctly for the ai-proxy plugin's architecture. The plugin expects drivers to use headers passed via extra_opts.headers (see openai-base.lua:244-245), but this driver creates its own headers. The api_key from conf.api_key needs to be properly integrated with the auth headers system used by the base plugin (see base.lua:64 where auth.header is passed).
| APISIX automatically performs "protocol translation" when using the Anthropic provider. This ensures that your existing OpenAI-compatible applications can switch to Claude models without any code modifications. | ||
|
|
||
| ### Protocol Translation Details | ||
|
|
||
| 1. **System Prompt Handling**: OpenAI embeds system instructions within the \`messages\` array. APISIX automatically extracts these and maps them to Anthropic's mandatory top-level \`system\` field. | ||
| 2. **Header Adaptation**: | ||
| - Translates \`Authorization: Bearer <key>\` to \`x-api-key: <key>\`. | ||
| - Automatically injects the \`anthropic-version: 2023-06-01\` header. | ||
| 3. **Response Conversion**: Anthropic's response format is converted back to the OpenAI-compatible structure, including token usage statistics. | ||
|
|
There was a problem hiding this comment.
The documentation states that APISIX performs "protocol translation" from OpenAI format to Anthropic format, but the actual implementation in anthropic.lua does not properly handle this conversion. The driver's transform_request method copies message content as-is (line 58) without converting from OpenAI's string content format to Anthropic's array-of-objects format. This documentation is inconsistent with the implementation.
| -- 4. Validate Anthropic's native message structure | ||
| -- Messages must have content as array with type field | ||
| local msg = body.messages[1] | ||
| if type(msg.content) ~= "table" | ||
| or msg.content[1].type ~= "text" then | ||
| ngx.status = 400 | ||
| ngx.say("invalid anthropic message format") | ||
| return | ||
| end |
There was a problem hiding this comment.
The test mock server validates the Anthropic native message format (lines 61-67), expecting content to be a table with a type field. However, there's no code in the Anthropic driver that performs this transformation. The driver would need to convert OpenAI's string-based content to Anthropic's array-based content structure, but the transform_request method at line 58 just copies content as-is. This test would fail with an actual OpenAI-format input.
| curl http://127.0.0.1:9080/v1/messages -X POST \ | ||
| -H "Content-Type: application/json" \ | ||
| -d '{ | ||
| "model": "gpt-4", |
There was a problem hiding this comment.
The documentation example specifies model as "gpt-4" (line 75), which is an OpenAI model identifier, not an Anthropic model. According to the attributes table (line 24), the model should be an Anthropic model ID like "claude-3-5-sonnet-20240620". Using an OpenAI model name in the example is confusing and would likely fail with the actual Anthropic API.
| "model": "gpt-4", | |
| "model": "claude-3-5-sonnet-20240620", |
| function _M:request(url, body, headers, timeout) | ||
| local httpc = http.new( ) | ||
| if timeout then | ||
| httpc:set_timeout(timeout ) | ||
| end | ||
|
|
||
| local res, err = httpc:request_uri(url, { | ||
| method = "POST", | ||
| body = core.json.encode(body ), | ||
| headers = headers, | ||
| ssl_verify = false, | ||
| }) | ||
|
|
||
| if not res then | ||
| return nil, "failed to request AI provider: " .. err | ||
| end | ||
|
|
There was a problem hiding this comment.
The ai-driver-base.lua provides a simple request method, but this is insufficient for AI driver implementations. The method lacks critical functionality that drivers need: 1) streaming response handling (SSE), 2) context variable updates for logging/metrics, 3) response filtering hooks, 4) token usage extraction, and 5) proper error handling. Looking at openai-base.lua:196-290, a proper AI driver request method requires significantly more complexity. This base implementation cannot serve as a functional parent for AI drivers.
| function _M:request(url, body, headers, timeout) | |
| local httpc = http.new( ) | |
| if timeout then | |
| httpc:set_timeout(timeout ) | |
| end | |
| local res, err = httpc:request_uri(url, { | |
| method = "POST", | |
| body = core.json.encode(body ), | |
| headers = headers, | |
| ssl_verify = false, | |
| }) | |
| if not res then | |
| return nil, "failed to request AI provider: " .. err | |
| end | |
| -- Generic request helper for AI drivers. | |
| -- Supports: | |
| -- * Optional streaming responses (SSE-style) via opts.stream / opts.on_chunk | |
| -- * Context updates for logging/metrics via opts.ctx or self.ctx | |
| -- * Response filtering hooks via opts.filter_response or self.filter_response | |
| -- * Optional token usage extraction via opts.extract_usage or self.extract_usage | |
| -- * Improved error handling and status code checking | |
| function _M:request(url, body, headers, timeout, opts) | |
| opts = opts or {} | |
| headers = headers or {} | |
| local ctx = opts.ctx or self.ctx | |
| local filter_response = opts.filter_response or self.filter_response | |
| local extract_usage = opts.extract_usage or self.extract_usage | |
| local on_chunk = opts.on_chunk or self.on_chunk | |
| local stream = opts.stream == true | |
| local httpc, err = http.new() | |
| if not httpc then | |
| return nil, "failed to create HTTP client for AI provider: " .. (err or "unknown error") | |
| end | |
| local effective_timeout = timeout or opts.timeout | |
| if effective_timeout then | |
| httpc:set_timeout(effective_timeout) | |
| end | |
| local encoded_body, encode_err = core.json.encode(body) | |
| if not encoded_body then | |
| return nil, "failed to encode request body as JSON: " .. (encode_err or "unknown error") | |
| end | |
| if ctx then | |
| ctx.bytes_sent = (ctx.bytes_sent or 0) + #encoded_body | |
| end | |
| local params = { | |
| method = "POST", | |
| body = encoded_body, | |
| headers = headers, | |
| ssl_verify = false, | |
| } | |
| if stream then | |
| params.stream = true | |
| params.keepalive = false | |
| end | |
| local res, req_err = httpc:request_uri(url, params) | |
| if not res then | |
| return nil, "failed to request AI provider: " .. (req_err or "unknown error") | |
| end | |
| if ctx then | |
| ctx.last_status = res.status | |
| end | |
| -- Treat HTTP 4xx/5xx as errors for callers, but still expose details. | |
| if res.status >= 400 then | |
| local msg = "AI provider returned HTTP status " .. tostring(res.status) | |
| if res.body and res.body ~= "" then | |
| msg = msg .. ": " .. res.body | |
| end | |
| return nil, msg | |
| end | |
| -- Streaming mode: read chunks and feed them to callbacks / filters. | |
| if stream then | |
| local reader = res.body_reader | |
| if not reader then | |
| return nil, "streaming requested but no body_reader available from AI provider response" | |
| end | |
| while true do | |
| local chunk, read_err = reader(8192) | |
| if not chunk then | |
| if read_err and read_err ~= "closed" then | |
| return nil, "failed to read AI provider stream: " .. read_err | |
| end | |
| break | |
| end | |
| if chunk ~= "" then | |
| if ctx then | |
| ctx.bytes_received = (ctx.bytes_received or 0) + #chunk | |
| end | |
| if filter_response and type(filter_response) == "function" then | |
| local filtered, ferr = filter_response(chunk, ctx) | |
| if ferr then | |
| return nil, "response filter error: " .. tostring(ferr) | |
| end | |
| chunk = filtered or "" | |
| end | |
| if on_chunk and type(on_chunk) == "function" and chunk ~= "" then | |
| local ok_cb, cb_err = pcall(on_chunk, chunk, ctx) | |
| if not ok_cb then | |
| return nil, "on_chunk callback failed: " .. tostring(cb_err) | |
| end | |
| end | |
| end | |
| end | |
| -- For streaming, return the response metadata but without relying on body. | |
| res.body = nil | |
| return res | |
| end | |
| -- Non-streaming response handling. | |
| if res.body and ctx then | |
| ctx.bytes_received = (ctx.bytes_received or 0) + #res.body | |
| end | |
| -- Apply optional filtering on the full body. | |
| if filter_response and type(filter_response) == "function" and res.body then | |
| local filtered, ferr = filter_response(res.body, ctx) | |
| if ferr then | |
| return nil, "response filter error: " .. tostring(ferr) | |
| end | |
| res.body = filtered or res.body | |
| end | |
| -- Optional token usage extraction from JSON body. | |
| if extract_usage and type(extract_usage) == "function" and res.body then | |
| local ok_dec, decoded = pcall(core.json.decode, res.body) | |
| if ok_dec and decoded then | |
| local ok_ext, usage = pcall(extract_usage, decoded, ctx) | |
| if ok_ext and usage and ctx then | |
| ctx.token_usage = (ctx.token_usage or 0) + (usage or 0) | |
| end | |
| end | |
| end |
| local mt = { __index = setmetatable(_M, { __index = base }) } | ||
|
|
||
| local ANTHROPIC_VERSION = "2023-06-01" | ||
| local FINISH_REASON_MAP = { | ||
| ["end_turn"] = "stop", | ||
| ["max_tokens"] = "length", | ||
| ["stop_sequence"] = "stop", | ||
| ["tool_use"] = "tool_calls", | ||
| } | ||
|
|
||
| function _M.new(opts) | ||
| return setmetatable(opts or {}, mt) | ||
| end |
There was a problem hiding this comment.
The metatable setup creates a chain where _M.__index points to a table with __index = base (line 29). This means instances will first look in _M, then in base. However, this doesn't properly inherit from base because _M methods don't call parent methods and base.new is never used. The correct pattern for inheritance would be to either: 1) have _M.new call base.new and properly set up the metatable chain, or 2) directly inherit like openai.lua does by returning an instance from openai-base.new().
|
|
||
| \`\`\`json | ||
| { | ||
| "uri": "/v1/messages", |
There was a problem hiding this comment.
The documentation describes the uri as set to "/v1/messages" (line 48), but this is Anthropic's endpoint path, not the APISIX route URI. For the ai-proxy plugin, the URI typically matches the client's request path (e.g., "/v1/chat/completions" for OpenAI compatibility). The override.endpoint should specify the full Anthropic URL. This example configuration would not work as described.
| for _, msg in ipairs(request_table.messages) do | ||
| if msg.role == "system" then | ||
| anthropic_body.system = msg.content | ||
| else | ||
| core.table.insert(anthropic_body.messages, { | ||
| role = msg.role, | ||
| content = msg.content | ||
| }) | ||
| end | ||
| end |
There was a problem hiding this comment.
The transform_request method doesn't validate that request_table.messages exists or is non-empty before iterating over it (line 52). If called with an invalid request, this would cause a runtime error. The validate_request method should catch this, but since that method is missing from this driver, there's no validation happening.
| local base = require("apisix.plugins.ai-drivers.ai-driver-base") | ||
| local core = require("apisix.core") | ||
| local setmetatable = setmetatable | ||
|
|
||
| local _M = { | ||
| name = "anthropic", | ||
| host = "api.anthropic.com", | ||
| path = "/v1/messages", | ||
| port = 443, | ||
| } | ||
|
|
||
| local mt = { __index = setmetatable(_M, { __index = base }) } | ||
|
|
||
| local ANTHROPIC_VERSION = "2023-06-01" | ||
| local FINISH_REASON_MAP = { | ||
| ["end_turn"] = "stop", | ||
| ["max_tokens"] = "length", | ||
| ["stop_sequence"] = "stop", | ||
| ["tool_use"] = "tool_calls", | ||
| } | ||
|
|
||
| function _M.new(opts) | ||
| return setmetatable(opts or {}, mt) | ||
| end | ||
|
|
||
| function _M:transform_request(conf, request_table) | ||
| local anthropic_body = { | ||
| model = conf.model, | ||
| messages = {}, | ||
| max_tokens = request_table.max_tokens or 1024, | ||
| stream = request_table.stream, | ||
| } | ||
| ) | ||
|
|
||
| -- Protocol Translation: Extract system prompt | ||
| for _, msg in ipairs(request_table.messages) do | ||
| if msg.role == "system" then | ||
| anthropic_body.system = msg.content | ||
| else | ||
| core.table.insert(anthropic_body.messages, { | ||
| role = msg.role, | ||
| content = msg.content | ||
| }) | ||
| end | ||
| end | ||
|
|
||
| local headers = { | ||
| ["Content-Type"] = "application/json", | ||
| ["x-api-key"] = conf.api_key, | ||
| ["anthropic-version"] = ANTHROPIC_VERSION, | ||
| } | ||
|
|
||
| return anthropic_body, headers | ||
| end | ||
|
|
||
| function _M:transform_response(response_body) | ||
| local body = core.json.decode(response_body) | ||
| if not body or not body.content then | ||
| return nil, "invalid response from anthropic" | ||
| end | ||
|
|
||
| return { | ||
| id = body.id, | ||
| object = "chat.completion", | ||
| created = os.time(), | ||
| model = body.model, | ||
| choices = { | ||
| { | ||
| index = 0, | ||
| message = { | ||
| role = "assistant", | ||
| content = body.content[1].text, | ||
| }, | ||
| finish_reason = FINISH_REASON_MAP[body.stop_reason] or "stop" | ||
| } | ||
| }, | ||
| usage = { | ||
| prompt_tokens = body.usage.input_tokens, | ||
| completion_tokens = body.usage.output_tokens, | ||
| total_tokens = body.usage.input_tokens + body.usage.output_tokens | ||
| } | ||
| } | ||
| end | ||
|
|
||
| return _M |
There was a problem hiding this comment.
The Anthropic driver is missing the required validate_request method that is called by the ai-proxy base plugin (see apisix/plugins/ai-proxy/base.lua:56). This method must be implemented to validate incoming requests. Looking at the openai-base.lua driver (lines 56-68), the method should validate Content-Type headers and decode the JSON request body.
| local res, err = httpc:request_uri(url, { | ||
| method = "POST", | ||
| body = core.json.encode(body ), | ||
| headers = headers, | ||
| ssl_verify = false, |
There was a problem hiding this comment.
The request helper unconditionally sets ssl_verify = false when calling httpc:request_uri, which disables TLS certificate validation for all outbound AI provider requests. This allows any on-path attacker (e.g., via DNS poisoning or network MITM) to impersonate the Anthropic endpoint and capture prompts, responses, and API keys. Update this logic to validate TLS certificates by default (using ssl_verify = true or equivalent) and only allow disabling verification explicitly in tightly controlled test or debug configurations.
| local res, err = httpc:request_uri(url, { | |
| method = "POST", | |
| body = core.json.encode(body ), | |
| headers = headers, | |
| ssl_verify = false, | |
| -- Enable TLS certificate verification by default; allow explicit opt-out | |
| -- via self.ssl_verify == false for tightly controlled test/debug setups. | |
| local ssl_verify = true | |
| if self and self.ssl_verify == false then | |
| ssl_verify = false | |
| end | |
| local res, err = httpc:request_uri(url, { | |
| method = "POST", | |
| body = core.json.encode(body ), | |
| headers = headers, | |
| ssl_verify = ssl_verify, |
1 participant
Description
This PR introduces native Anthropic API support to the
ai-proxyplugin.Instead of relying on the OpenAI-compatible adapter, this implementation
adds first-class support for Anthropic’s native
/v1/messagesendpointand protocol, including request/response translation and streaming semantics.
Anthropic’s API is not fully compatible with OpenAI’s Chat Completions API:
it differs in endpoint shape, authentication headers, request/response schema,
and streaming behavior. Native support is required to ensure correctness,
extensibility, and long-term maintainability.
Which issue(s) this PR fixes:
This PR does not directly close an existing issue.
It addresses the gap in native Anthropic support discussed in previous reviews
and supersedes the earlier attempt in #12924, which was closed after branch
recreation during refactoring and test stabilization.
Changes introduced in this PR
ai-proxy/v1/messagesendpointx-api-keyanthropic-versionheaderBackward compatibility
This change is fully backward compatible.
Existing OpenAI-compatible configurations remain unchanged.
The native Anthropic driver is only enabled when the
anthropicprovider is explicitly configured.
Checklist