Skip to content

fix(middleware): parse OpenAI-spec tool_choice in /v1/chat/completions#9559

Merged
mudler merged 4 commits into
mudler:masterfrom
Anai-Guo:fix-toolchoice-chatcompletions
May 13, 2026
Merged

fix(middleware): parse OpenAI-spec tool_choice in /v1/chat/completions#9559
mudler merged 4 commits into
mudler:masterfrom
Anai-Guo:fix-toolchoice-chatcompletions

Conversation

@Anai-Guo
Copy link
Copy Markdown
Contributor

Summary

Closes the fourth-site clause in #9508. This is the natural follow-up to #9526 (merged): #9526 fixed the wrong-setter pattern at three endpoints, and walcz-de's #9509 (merged) fixed the parallel /v1/responses parsing. The remaining piece — tool_choice parsing in /v1/chat/completions (SetModelAndConfig middleware) — silently mishandled both the string-mode and the OpenAI-spec map shape.

The bug

core/http/middleware/request.go:322-335 (before this PR) unmarshalled every tool_choice shape through json.Unmarshal(..., &functions.Tool{}):

switch content := input.ToolsChoice.(type) {
case string:
    _ = json.Unmarshal([]byte(content), &toolChoice)
case map[string]any:
    dat, _ := json.Marshal(content)
    _ = json.Unmarshal(dat, &toolChoice)
}
input.FunctionCall = map[string]any{
    "name": toolChoice.Function.Name,
}

Two failure modes, both silenced via _ =:

  1. String mode (tool_choice: "required" | "auto" | "none") — json.Unmarshal([]byte("required"), &functions.Tool{}) fails. toolChoice.Function.Name == "". The downstream dispatch (lines 487+ in this same file) then runs SetFunctionCallNameString(""), so the requested mode never engages.
  2. OpenAI-spec map shape ({"type":"function", "function":{"name":"..."}}) — the JSON keys do not match functions.Tool's field tags (Tool nests function differently), so name again ends up empty.

The fix

Mirrors the parsing pattern walcz-de established in #9509 for MergeOpenResponsesConfig, dispatching by the actual incoming shape rather than reinterpreting through the wrong struct:

  • String case: if tool_choice is "required" / "none", hand it to input.FunctionCall as a string. The existing string-case branch downstream (line 487) routes that through SetFunctionCallString, which writes the mode field. "auto" is the default and skipped. Empty string is skipped.
  • Map case: accept both the OpenAI-spec nested shape ({"type":"function", "function":{"name":"..."}}) and the legacy/Anthropic-compat flat shape ({"type":"function", "name":"..."}). Nested is preferred; flat is the fallback. Set input.FunctionCall = {"name": <name>} only when a non-empty name was actually found, so the downstream map-case branch writes the name via SetFunctionCallNameString.

This way the existing dispatch on input.FunctionCall (lines 487-502) keeps doing what it already does correctly — the bug was strictly in the upstream parsing that fed it bad data.

The pkg/functions import drops with this change (it was only used for the now-removed var toolChoice functions.Tool).

Net effect (matches what #9509 + #9526 already restored elsewhere)

Request Before After
tool_choice: "required" SetFunctionCallNameString("") (silent no-op) SetFunctionCallString("required") engages mode
tool_choice: {type:"function", function:{name:"X"}} SetFunctionCallNameString("") (silent no-op) SetFunctionCallNameString("X") engages grammar forcing
tool_choice: {type:"function", name:"X"} (legacy) already worked unchanged
tool_choice: "auto" / "none" reached downstream as "" (no-op) skipped at parse time (no-op) — same observable behavior

Why no new tests in this PR

The parsing logic here is intentionally identical (modulo the string-vs-map dispatch difference dictated by the surrounding code) to MergeOpenResponsesConfig in #9509, which already has nine Ginkgo specs (MergeOpenResponsesConfig tool_choice parsing in request_test.go) covering all four shapes plus malformed inputs. Adding a parallel set for SetModelAndConfig would require a heavier scaffold (the existing tests there go through Echo + a real model dir) for coverage that's structurally redundant with the merged tests. Happy to add a unit-level test scaffold for SetModelAndConfig's parsing if a maintainer prefers — let me know.

Test plan

  • go build ./...
  • go test ./core/http/middleware/ -count=1 -run TestMiddleware
  • Send tool_choice: "required" to /v1/chat/completions with a tools array → grammar forcing engages, output arrives as tool_calls not free-text.
  • Send tool_choice: {"type":"function", "function":{"name":"my_tool"}}my_tool is forced.
  • Send tool_choice: {"type":"function", "name":"my_tool"} (legacy) → still works.

🤖 Generated with Claude Code

mudler added a commit to Anai-Guo/LocalAI that referenced this pull request May 13, 2026
…oded tool_choice

Some non-spec clients send tool_choice as a JSON-encoded string of an
object form, e.g. "{\"type\":\"function\",\"function\":{\"name\":\"X\"}}".
The pre-mudler#9559 code accepted this by accident: its case string: branch
ran json.Unmarshal([]byte(content), &functions.Tool{}), which succeeded
for that double-encoded shape even though it failed for the legitimate
plain string modes "auto" / "none" / "required".

The first version of this PR routed every string straight to
SetFunctionCallString as a mode, which fixed the plain-string cases but
silently regressed the double-encoded one (funcs.Select("{...}") returns
nothing). Restore the fallback: when a string looks like a JSON object,
try parsing it as a tool_choice map first; fall through to mode-string
handling only when no usable name comes out.

Factor the map-name extraction into a small helper
(extractToolChoiceFunctionName) so the string-fallback and the regular
map case go through identical code, and accept both the OpenAI-spec
nested shape and the legacy/Anthropic flat shape from either entry
point.

Add 3 Ginkgo specs covering the double-encoded case (nested form, legacy
form, and the fall-through when the JSON has no usable name).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
Anai-Guo and others added 4 commits May 13, 2026 20:49
Follows up on mudler#9526 (the 3-site setter fix) by addressing the remaining
clause in mudler#9508 — string mode and OpenAI-spec specific-function shape both
silently failed in the /v1/chat/completions parsing path.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…h specs

The previous commit on this branch saved core/http/middleware/request.go
with CRLF line endings, ballooning the diff against master to 684 / 651
for what is in reality a ~50-line parsing change. Restore LF (matches
.editorconfig end_of_line = lf).

Add 11 Ginkgo specs under "SetModelAndConfig tool_choice parsing
(chat completions)" that parallel the existing MergeOpenResponsesConfig
specs from mudler#9509. They drive the full middleware chain (SetModelAndConfig
+ SetOpenAIRequest) and assert:

  * "required"  -> ShouldUseFunctions=true, no specific name
  * "none"      -> ShouldUseFunctions=false (tools disabled per OpenAI spec)
  * "auto"      -> default, tools available, no specific name
  * {type:function, function:{name:X}}  (spec)    -> X is forced
  * {type:function, name:X}             (legacy)  -> X is forced
  * nested wins when both forms are present
  * malformed shapes (no type, wrong type, no name, empty name) are no-ops

Update the inline comment on the string case to describe the actual
mechanism: "none" reaches SetFunctionCallString("none") downstream and
is then honored by ShouldUseFunctions() returning false. Before this PR
json.Unmarshal([]byte("none"), &functions.Tool{}) failed silently, so
"none" was ignored - making "none" actually work is a real behavior fix
this PR brings.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
…oded tool_choice

Some non-spec clients send tool_choice as a JSON-encoded string of an
object form, e.g. "{\"type\":\"function\",\"function\":{\"name\":\"X\"}}".
The pre-mudler#9559 code accepted this by accident: its case string: branch
ran json.Unmarshal([]byte(content), &functions.Tool{}), which succeeded
for that double-encoded shape even though it failed for the legitimate
plain string modes "auto" / "none" / "required".

The first version of this PR routed every string straight to
SetFunctionCallString as a mode, which fixed the plain-string cases but
silently regressed the double-encoded one (funcs.Select("{...}") returns
nothing). Restore the fallback: when a string looks like a JSON object,
try parsing it as a tool_choice map first; fall through to mode-string
handling only when no usable name comes out.

Factor the map-name extraction into a small helper
(extractToolChoiceFunctionName) so the string-fallback and the regular
map case go through identical code, and accept both the OpenAI-spec
nested shape and the legacy/Anthropic flat shape from either entry
point.

Add 3 Ginkgo specs covering the double-encoded case (nested form, legacy
form, and the fall-through when the JSON has no usable name).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
The new tool_choice parsing tests added a second AfterEach that calls
os.RemoveAll(modelDir) without checking the error; errcheck flagged it.
Suppress with the standard _ = idiom. The pre-existing AfterEach on the
earlier Describe still elides the check the same way it did before -
leaving that untouched to keep this commit minimal.

Assisted-by: Claude:opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler force-pushed the fix-toolchoice-chatcompletions branch from dfa761f to 96e5dfb Compare May 13, 2026 21:00
@mudler mudler merged commit 67c34bb into mudler:master May 13, 2026
55 checks passed
@mudler mudler added bug Something isn't working labels May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready-to-merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants