fix(middleware): parse OpenAI-spec tool_choice in /v1/chat/completions#9559
Merged
Conversation
mudler
added a commit
to Anai-Guo/LocalAI
that referenced
this pull request
May 13, 2026
…oded tool_choice
Some non-spec clients send tool_choice as a JSON-encoded string of an
object form, e.g. "{\"type\":\"function\",\"function\":{\"name\":\"X\"}}".
The pre-mudler#9559 code accepted this by accident: its case string: branch
ran json.Unmarshal([]byte(content), &functions.Tool{}), which succeeded
for that double-encoded shape even though it failed for the legitimate
plain string modes "auto" / "none" / "required".
The first version of this PR routed every string straight to
SetFunctionCallString as a mode, which fixed the plain-string cases but
silently regressed the double-encoded one (funcs.Select("{...}") returns
nothing). Restore the fallback: when a string looks like a JSON object,
try parsing it as a tool_choice map first; fall through to mode-string
handling only when no usable name comes out.
Factor the map-name extraction into a small helper
(extractToolChoiceFunctionName) so the string-fallback and the regular
map case go through identical code, and accept both the OpenAI-spec
nested shape and the legacy/Anthropic flat shape from either entry
point.
Add 3 Ginkgo specs covering the double-encoded case (nested form, legacy
form, and the fall-through when the JSON has no usable name).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
Follows up on mudler#9526 (the 3-site setter fix) by addressing the remaining clause in mudler#9508 — string mode and OpenAI-spec specific-function shape both silently failed in the /v1/chat/completions parsing path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…h specs The previous commit on this branch saved core/http/middleware/request.go with CRLF line endings, ballooning the diff against master to 684 / 651 for what is in reality a ~50-line parsing change. Restore LF (matches .editorconfig end_of_line = lf). Add 11 Ginkgo specs under "SetModelAndConfig tool_choice parsing (chat completions)" that parallel the existing MergeOpenResponsesConfig specs from mudler#9509. They drive the full middleware chain (SetModelAndConfig + SetOpenAIRequest) and assert: * "required" -> ShouldUseFunctions=true, no specific name * "none" -> ShouldUseFunctions=false (tools disabled per OpenAI spec) * "auto" -> default, tools available, no specific name * {type:function, function:{name:X}} (spec) -> X is forced * {type:function, name:X} (legacy) -> X is forced * nested wins when both forms are present * malformed shapes (no type, wrong type, no name, empty name) are no-ops Update the inline comment on the string case to describe the actual mechanism: "none" reaches SetFunctionCallString("none") downstream and is then honored by ShouldUseFunctions() returning false. Before this PR json.Unmarshal([]byte("none"), &functions.Tool{}) failed silently, so "none" was ignored - making "none" actually work is a real behavior fix this PR brings. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4-7 [Claude Code]
…oded tool_choice
Some non-spec clients send tool_choice as a JSON-encoded string of an
object form, e.g. "{\"type\":\"function\",\"function\":{\"name\":\"X\"}}".
The pre-mudler#9559 code accepted this by accident: its case string: branch
ran json.Unmarshal([]byte(content), &functions.Tool{}), which succeeded
for that double-encoded shape even though it failed for the legitimate
plain string modes "auto" / "none" / "required".
The first version of this PR routed every string straight to
SetFunctionCallString as a mode, which fixed the plain-string cases but
silently regressed the double-encoded one (funcs.Select("{...}") returns
nothing). Restore the fallback: when a string looks like a JSON object,
try parsing it as a tool_choice map first; fall through to mode-string
handling only when no usable name comes out.
Factor the map-name extraction into a small helper
(extractToolChoiceFunctionName) so the string-fallback and the regular
map case go through identical code, and accept both the OpenAI-spec
nested shape and the legacy/Anthropic flat shape from either entry
point.
Add 3 Ginkgo specs covering the double-encoded case (nested form, legacy
form, and the fall-through when the JSON has no usable name).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
The new tool_choice parsing tests added a second AfterEach that calls os.RemoveAll(modelDir) without checking the error; errcheck flagged it. Suppress with the standard _ = idiom. The pre-existing AfterEach on the earlier Describe still elides the check the same way it did before - leaving that untouched to keep this commit minimal. Assisted-by: Claude:opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
dfa761f to
96e5dfb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the fourth-site clause in #9508. This is the natural follow-up to #9526 (merged): #9526 fixed the wrong-setter pattern at three endpoints, and walcz-de's #9509 (merged) fixed the parallel
/v1/responsesparsing. The remaining piece — tool_choice parsing in/v1/chat/completions(SetModelAndConfigmiddleware) — silently mishandled both the string-mode and the OpenAI-spec map shape.The bug
core/http/middleware/request.go:322-335(before this PR) unmarshalled every tool_choice shape throughjson.Unmarshal(..., &functions.Tool{}):Two failure modes, both silenced via
_ =:tool_choice: "required" | "auto" | "none") —json.Unmarshal([]byte("required"), &functions.Tool{})fails.toolChoice.Function.Name == "". The downstream dispatch (lines 487+ in this same file) then runsSetFunctionCallNameString(""), so the requested mode never engages.{"type":"function", "function":{"name":"..."}}) — the JSON keys do not matchfunctions.Tool's field tags (Tool nestsfunctiondifferently), sonameagain ends up empty.The fix
Mirrors the parsing pattern walcz-de established in #9509 for
MergeOpenResponsesConfig, dispatching by the actual incoming shape rather than reinterpreting through the wrong struct:tool_choiceis"required"/"none", hand it toinput.FunctionCallas a string. The existing string-case branch downstream (line 487) routes that throughSetFunctionCallString, which writes the mode field."auto"is the default and skipped. Empty string is skipped.{"type":"function", "function":{"name":"..."}}) and the legacy/Anthropic-compat flat shape ({"type":"function", "name":"..."}). Nested is preferred; flat is the fallback. Setinput.FunctionCall = {"name": <name>}only when a non-empty name was actually found, so the downstream map-case branch writes the name viaSetFunctionCallNameString.This way the existing dispatch on
input.FunctionCall(lines 487-502) keeps doing what it already does correctly — the bug was strictly in the upstream parsing that fed it bad data.The
pkg/functionsimport drops with this change (it was only used for the now-removedvar toolChoice functions.Tool).Net effect (matches what #9509 + #9526 already restored elsewhere)
tool_choice: "required"SetFunctionCallNameString("")(silent no-op)SetFunctionCallString("required")engages modetool_choice: {type:"function", function:{name:"X"}}SetFunctionCallNameString("")(silent no-op)SetFunctionCallNameString("X")engages grammar forcingtool_choice: {type:"function", name:"X"}(legacy)tool_choice: "auto"/"none"""(no-op)Why no new tests in this PR
The parsing logic here is intentionally identical (modulo the string-vs-map dispatch difference dictated by the surrounding code) to
MergeOpenResponsesConfigin #9509, which already has nine Ginkgo specs (MergeOpenResponsesConfig tool_choice parsinginrequest_test.go) covering all four shapes plus malformed inputs. Adding a parallel set forSetModelAndConfigwould require a heavier scaffold (the existing tests there go through Echo + a real model dir) for coverage that's structurally redundant with the merged tests. Happy to add a unit-level test scaffold forSetModelAndConfig's parsing if a maintainer prefers — let me know.Test plan
go build ./...go test ./core/http/middleware/ -count=1 -run TestMiddlewaretool_choice: "required"to/v1/chat/completionswith a tools array → grammar forcing engages, output arrives astool_callsnot free-text.tool_choice: {"type":"function", "function":{"name":"my_tool"}}→my_toolis forced.tool_choice: {"type":"function", "name":"my_tool"}(legacy) → still works.🤖 Generated with Claude Code