Skip to content

Commit 67c34bb

Browse files
Anai-Guomudler
Anai-Guo
and
mudler
authored
fix(middleware): parse OpenAI-spec tool_choice in /v1/chat/completions (#9559)
* fix(middleware): parse OpenAI-spec tool_choice in /v1/chat/completions Follows up on #9526 (the 3-site setter fix) by addressing the remaining clause in #9508 — string mode and OpenAI-spec specific-function shape both silently failed in the /v1/chat/completions parsing path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(middleware): restore LF endings and cover tool_choice parsing with specs The previous commit on this branch saved core/http/middleware/request.go with CRLF line endings, ballooning the diff against master to 684 / 651 for what is in reality a ~50-line parsing change. Restore LF (matches .editorconfig end_of_line = lf). Add 11 Ginkgo specs under "SetModelAndConfig tool_choice parsing (chat completions)" that parallel the existing MergeOpenResponsesConfig specs from #9509. They drive the full middleware chain (SetModelAndConfig + SetOpenAIRequest) and assert: * "required" -> ShouldUseFunctions=true, no specific name * "none" -> ShouldUseFunctions=false (tools disabled per OpenAI spec) * "auto" -> default, tools available, no specific name * {type:function, function:{name:X}} (spec) -> X is forced * {type:function, name:X} (legacy) -> X is forced * nested wins when both forms are present * malformed shapes (no type, wrong type, no name, empty name) are no-ops Update the inline comment on the string case to describe the actual mechanism: "none" reaches SetFunctionCallString("none") downstream and is then honored by ShouldUseFunctions() returning false. Before this PR json.Unmarshal([]byte("none"), &functions.Tool{}) failed silently, so "none" was ignored - making "none" actually work is a real behavior fix this PR brings. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4-7 [Claude Code] * fix(middleware): preserve pre-#9559 support for JSON-string-encoded tool_choice Some non-spec clients send tool_choice as a JSON-encoded string of an object form, e.g. "{\"type\":\"function\",\"function\":{\"name\":\"X\"}}". The pre-#9559 code accepted this by accident: its case string: branch ran json.Unmarshal([]byte(content), &functions.Tool{}), which succeeded for that double-encoded shape even though it failed for the legitimate plain string modes "auto" / "none" / "required". The first version of this PR routed every string straight to SetFunctionCallString as a mode, which fixed the plain-string cases but silently regressed the double-encoded one (funcs.Select("{...}") returns nothing). Restore the fallback: when a string looks like a JSON object, try parsing it as a tool_choice map first; fall through to mode-string handling only when no usable name comes out. Factor the map-name extraction into a small helper (extractToolChoiceFunctionName) so the string-fallback and the regular map case go through identical code, and accept both the OpenAI-spec nested shape and the legacy/Anthropic flat shape from either entry point. Add 3 Ginkgo specs covering the double-encoded case (nested form, legacy form, and the fall-through when the JSON has no usable name). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4-7 [Claude Code] * test(middleware): silence errcheck on AfterEach os.RemoveAll The new tool_choice parsing tests added a second AfterEach that calls os.RemoveAll(modelDir) without checking the error; errcheck flagged it. Suppress with the standard _ = idiom. The pre-existing AfterEach on the earlier Describe still elides the check the same way it did before - leaving that untouched to keep this commit minimal. Assisted-by: Claude:opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1 parent 4430fae commit 67c34bb

2 files changed

Lines changed: 313 additions & 9 deletions

File tree

‎core/http/middleware/request.go‎

Lines changed: 68 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,6 @@ import (
1414
"github.com/mudler/LocalAI/core/schema"
1515
"github.com/mudler/LocalAI/core/services/galleryop"
1616
"github.com/mudler/LocalAI/core/templates"
17-
"github.com/mudler/LocalAI/pkg/functions"
1817
"github.com/mudler/LocalAI/pkg/model"
1918
"github.com/mudler/LocalAI/pkg/utils"
2019
"github.com/mudler/xlog"
@@ -241,6 +240,28 @@ func (re *RequestExtractor) SetOpenAIRequest(c echo.Context) error {
241240
return nil
242241
}
243242

243+
// extractToolChoiceFunctionName parses a tool_choice map and returns the
244+
// specific function name. Accepts both the OpenAI-spec nested shape
245+
// ({type:function, function:{name:...}}) and the legacy/Anthropic-compat
246+
// flat shape ({type:function, name:...}); the nested form wins when both
247+
// are present. Returns "" for malformed input or when the shape names a
248+
// mode rather than a specific tool.
249+
func extractToolChoiceFunctionName(m map[string]any) string {
250+
tcType, ok := m["type"].(string)
251+
if !ok || tcType != "function" {
252+
return ""
253+
}
254+
if fn, ok := m["function"].(map[string]any); ok {
255+
if n, ok := fn["name"].(string); ok && n != "" {
256+
return n
257+
}
258+
}
259+
if n, ok := m["name"].(string); ok {
260+
return n
261+
}
262+
return ""
263+
}
264+
244265
func mergeOpenAIRequestAndModelConfig(config *config.ModelConfig, input *schema.OpenAIRequest) error {
245266
if input.Echo {
246267
config.Echo = input.Echo
@@ -320,17 +341,55 @@ func mergeOpenAIRequestAndModelConfig(config *config.ModelConfig, input *schema.
320341
}
321342

322343
if input.ToolsChoice != nil {
323-
var toolChoice functions.Tool
324-
344+
// OpenAI tool_choice has three valid shapes plus one tolerated
345+
// non-spec form seen in the wild:
346+
//
347+
// 1. string mode: "auto" | "none" | "required"
348+
// 2. specific tool: {"type":"function", "function":{"name":"..."}} (current spec)
349+
// 3. legacy: {"type":"function", "name":"..."} (older / Anthropic-compat)
350+
// 4. double-encoded: "{\"type\":\"function\", ...}" (some clients serialize the object)
351+
//
352+
// The pre-#9559 code unmarshalled the string case through
353+
// json.Unmarshal([]byte(content), &functions.Tool{}), which:
354+
// - failed for plain string modes (so "required" / "none" were
355+
// silently ignored and tools stayed enabled regardless), but
356+
// - happened to handle shape 4 by accident.
357+
// It also could not parse shape 3 because functions.Tool has no
358+
// flat top-level Name field.
359+
//
360+
// Mirror the parsing pattern from MergeOpenResponsesConfig (#9509),
361+
// route results through the existing input.FunctionCall string/map
362+
// dispatch downstream (see the switch on input.FunctionCall in this
363+
// same function), and preserve the shape-4 fallback so non-spec
364+
// clients don't silently break. Tracked in #9508; sibling fix in #9526.
325365
switch content := input.ToolsChoice.(type) {
326366
case string:
327-
_ = json.Unmarshal([]byte(content), &toolChoice)
367+
// "auto" is the default and needs no override. "none" and "required"
368+
// both reach SetFunctionCallString via the input.FunctionCall string
369+
// branch below; ShouldUseFunctions() then returns false for "none"
370+
// (tools disabled) and true for "required" (mode engaged).
371+
//
372+
// If the string looks like a JSON object, try shape 4 first: parse
373+
// it as a tool_choice map and use the resulting name. Falling back
374+
// to mode-string handling when the parse yields no usable name keeps
375+
// genuinely-malformed input from accidentally engaging a mode.
376+
if content == "" || content == "auto" {
377+
break
378+
}
379+
if strings.HasPrefix(strings.TrimSpace(content), "{") {
380+
var nested map[string]any
381+
if err := json.Unmarshal([]byte(content), &nested); err == nil {
382+
if name := extractToolChoiceFunctionName(nested); name != "" {
383+
input.FunctionCall = map[string]any{"name": name}
384+
break
385+
}
386+
}
387+
}
388+
input.FunctionCall = content
328389
case map[string]any:
329-
dat, _ := json.Marshal(content)
330-
_ = json.Unmarshal(dat, &toolChoice)
331-
}
332-
input.FunctionCall = map[string]any{
333-
"name": toolChoice.Function.Name,
390+
if name := extractToolChoiceFunctionName(content); name != "" {
391+
input.FunctionCall = map[string]any{"name": name}
392+
}
334393
}
335394
}
336395

‎core/http/middleware/request_test.go‎

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,3 +306,248 @@ var _ = Describe("MergeOpenResponsesConfig tool_choice parsing", func() {
306306
})
307307
})
308308
})
309+
310+
// ---------------------------------------------------------------------------
311+
// SetModelAndConfig + SetOpenAIRequest - /v1/chat/completions tool_choice parsing
312+
// ---------------------------------------------------------------------------
313+
//
314+
// Parallel to the MergeOpenResponsesConfig specs above, but for the chat
315+
// completions path. The parsing block lives in mergeOpenAIRequestAndModelConfig
316+
// (called from SetOpenAIRequest), so these tests drive the full middleware
317+
// chain the way the production /v1/chat/completions route does.
318+
//
319+
// What we assert per shape:
320+
// - "required" -> ShouldUseFunctions=true, no specific name
321+
// - "none" -> ShouldUseFunctions=false (tools disabled)
322+
// - "auto" -> ShouldUseFunctions=true, no specific name
323+
// - {type:function, function:{name:"X"}} (spec) -> ShouldCallSpecificFunction=true, FunctionToCall="X"
324+
// - {type:function, name:"X"} (legacy) -> ShouldCallSpecificFunction=true, FunctionToCall="X"
325+
// - nested+flat both present -> nested wins
326+
// - malformed (no type / no name) -> no-op
327+
var _ = Describe("SetModelAndConfig tool_choice parsing (chat completions)", func() {
328+
var (
329+
app *echo.Echo
330+
modelDir string
331+
capturedConfig *config.ModelConfig
332+
)
333+
334+
BeforeEach(func() {
335+
var err error
336+
modelDir, err = os.MkdirTemp("", "localai-test-models-*")
337+
Expect(err).ToNot(HaveOccurred())
338+
339+
cfgContent := []byte("name: test-model\nbackend: llama-cpp\n")
340+
Expect(os.WriteFile(filepath.Join(modelDir, "test-model.yaml"), cfgContent, 0644)).To(Succeed())
341+
342+
ss := &system.SystemState{
343+
Model: system.Model{ModelsPath: modelDir},
344+
}
345+
appConfig := config.NewApplicationConfig()
346+
appConfig.SystemState = ss
347+
348+
mcl := config.NewModelConfigLoader(modelDir)
349+
ml := model.NewModelLoader(ss)
350+
re := NewRequestExtractor(mcl, ml, appConfig)
351+
352+
capturedConfig = nil
353+
app = echo.New()
354+
app.POST("/v1/chat/completions",
355+
func(c echo.Context) error {
356+
if cfg, ok := c.Get(CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig); ok {
357+
capturedConfig = cfg
358+
}
359+
return c.String(http.StatusOK, "ok")
360+
},
361+
re.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.OpenAIRequest) }),
362+
func(next echo.HandlerFunc) echo.HandlerFunc {
363+
return func(c echo.Context) error {
364+
if err := re.SetOpenAIRequest(c); err != nil {
365+
return err
366+
}
367+
return next(c)
368+
}
369+
},
370+
)
371+
})
372+
373+
AfterEach(func() {
374+
_ = os.RemoveAll(modelDir)
375+
})
376+
377+
// chatReq wraps a tool_choice JSON fragment in a minimal valid chat-completions
378+
// payload. The tools array is non-empty so downstream code paths that gate on
379+
// len(input.Functions) see something to work with.
380+
chatReq := func(toolChoiceJSON string) string {
381+
return `{"model":"test-model",` +
382+
`"messages":[{"role":"user","content":"hi"}],` +
383+
`"tools":[{"type":"function","function":{"name":"get_weather"}}],` +
384+
`"tool_choice":` + toolChoiceJSON + `}`
385+
}
386+
387+
Context("string tool_choice", func() {
388+
It("engages mode for tool_choice=\"required\"", func() {
389+
rec := postJSON(app, "/v1/chat/completions", chatReq(`"required"`))
390+
391+
Expect(rec.Code).To(Equal(http.StatusOK))
392+
Expect(capturedConfig).ToNot(BeNil())
393+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
394+
Expect(capturedConfig.ShouldUseFunctions()).To(BeTrue())
395+
})
396+
397+
It("disables tools for tool_choice=\"none\"", func() {
398+
// Before #9559 this was a silent no-op (json.Unmarshal of "none"
399+
// into functions.Tool failed); now "none" is honored per OpenAI spec.
400+
rec := postJSON(app, "/v1/chat/completions", chatReq(`"none"`))
401+
402+
Expect(rec.Code).To(Equal(http.StatusOK))
403+
Expect(capturedConfig).ToNot(BeNil())
404+
Expect(capturedConfig.ShouldUseFunctions()).To(BeFalse())
405+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
406+
})
407+
408+
It("leaves config untouched for tool_choice=\"auto\"", func() {
409+
rec := postJSON(app, "/v1/chat/completions", chatReq(`"auto"`))
410+
411+
Expect(rec.Code).To(Equal(http.StatusOK))
412+
Expect(capturedConfig).ToNot(BeNil())
413+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
414+
// "auto" is the default: tools available, model decides.
415+
Expect(capturedConfig.ShouldUseFunctions()).To(BeTrue())
416+
Expect(capturedConfig.FunctionToCall()).To(Equal(""))
417+
})
418+
})
419+
420+
Context("specific-function tool_choice (OpenAI spec shape)", func() {
421+
It("parses {type:function, function:{name:...}} and forces the named function", func() {
422+
rec := postJSON(app, "/v1/chat/completions",
423+
chatReq(`{"type":"function","function":{"name":"get_weather"}}`))
424+
425+
Expect(rec.Code).To(Equal(http.StatusOK))
426+
Expect(capturedConfig).ToNot(BeNil())
427+
// Key invariant: a correctly-formed OpenAI tool_choice must engage
428+
// grammar-based forcing via SetFunctionCallNameString.
429+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeTrue())
430+
Expect(capturedConfig.FunctionToCall()).To(Equal("get_weather"))
431+
})
432+
433+
It("prefers the nested function.name over a stray top-level name", func() {
434+
rec := postJSON(app, "/v1/chat/completions",
435+
chatReq(`{"type":"function","function":{"name":"correct_name"},"name":"legacy_name"}`))
436+
437+
Expect(rec.Code).To(Equal(http.StatusOK))
438+
Expect(capturedConfig).ToNot(BeNil())
439+
Expect(capturedConfig.FunctionToCall()).To(Equal("correct_name"))
440+
})
441+
})
442+
443+
Context("specific-function tool_choice (legacy Anthropic-compat shape)", func() {
444+
It("parses {type:function, name:...} and forces the named function", func() {
445+
rec := postJSON(app, "/v1/chat/completions",
446+
chatReq(`{"type":"function","name":"get_weather"}`))
447+
448+
Expect(rec.Code).To(Equal(http.StatusOK))
449+
Expect(capturedConfig).ToNot(BeNil())
450+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeTrue())
451+
Expect(capturedConfig.FunctionToCall()).To(Equal("get_weather"))
452+
})
453+
})
454+
455+
// Some non-spec clients send the object form serialized as a JSON string.
456+
// The pre-#9559 code accepted that by accident; this Context locks in
457+
// continued tolerance so those clients do not silently regress.
458+
Context("double-encoded tool_choice (JSON string of an object, non-spec)", func() {
459+
It("parses a serialized OpenAI-spec nested object", func() {
460+
// tool_choice value is itself a JSON-encoded string containing the
461+
// object form. Use json.Marshal of the inner blob so the escapes
462+
// are correct regardless of the test reader.
463+
inner := `{"type":"function","function":{"name":"get_weather"}}`
464+
encoded, err := json.Marshal(inner)
465+
Expect(err).ToNot(HaveOccurred())
466+
rec := postJSON(app, "/v1/chat/completions", chatReq(string(encoded)))
467+
468+
Expect(rec.Code).To(Equal(http.StatusOK))
469+
Expect(capturedConfig).ToNot(BeNil())
470+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeTrue())
471+
Expect(capturedConfig.FunctionToCall()).To(Equal("get_weather"))
472+
})
473+
474+
It("parses a serialized legacy/Anthropic flat object", func() {
475+
inner := `{"type":"function","name":"get_weather"}`
476+
encoded, err := json.Marshal(inner)
477+
Expect(err).ToNot(HaveOccurred())
478+
rec := postJSON(app, "/v1/chat/completions", chatReq(string(encoded)))
479+
480+
Expect(rec.Code).To(Equal(http.StatusOK))
481+
Expect(capturedConfig).ToNot(BeNil())
482+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeTrue())
483+
Expect(capturedConfig.FunctionToCall()).To(Equal("get_weather"))
484+
})
485+
486+
It("falls back to mode-string handling when the JSON string parses but has no usable name", func() {
487+
// A JSON-string that decodes to a map without a function name
488+
// should not engage specific-function forcing. We expect it to
489+
// fall through to the mode-string path; the resulting mode is
490+
// the raw blob (nonsense), but ShouldCallSpecificFunction stays
491+
// false - the invariant that matters.
492+
inner := `{"type":"function"}`
493+
encoded, err := json.Marshal(inner)
494+
Expect(err).ToNot(HaveOccurred())
495+
rec := postJSON(app, "/v1/chat/completions", chatReq(string(encoded)))
496+
497+
Expect(rec.Code).To(Equal(http.StatusOK))
498+
Expect(capturedConfig).ToNot(BeNil())
499+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
500+
})
501+
})
502+
503+
Context("malformed tool_choice", func() {
504+
It("is a no-op when type is missing", func() {
505+
rec := postJSON(app, "/v1/chat/completions",
506+
chatReq(`{"function":{"name":"get_weather"}}`))
507+
508+
Expect(rec.Code).To(Equal(http.StatusOK))
509+
Expect(capturedConfig).ToNot(BeNil())
510+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
511+
})
512+
513+
It("is a no-op when type is not \"function\"", func() {
514+
rec := postJSON(app, "/v1/chat/completions",
515+
chatReq(`{"type":"object","function":{"name":"get_weather"}}`))
516+
517+
Expect(rec.Code).To(Equal(http.StatusOK))
518+
Expect(capturedConfig).ToNot(BeNil())
519+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
520+
})
521+
522+
It("is a no-op when name is missing from both shapes", func() {
523+
rec := postJSON(app, "/v1/chat/completions",
524+
chatReq(`{"type":"function","function":{}}`))
525+
526+
Expect(rec.Code).To(Equal(http.StatusOK))
527+
Expect(capturedConfig).ToNot(BeNil())
528+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
529+
Expect(capturedConfig.FunctionToCall()).To(Equal(""))
530+
})
531+
532+
It("is a no-op when name is empty string", func() {
533+
rec := postJSON(app, "/v1/chat/completions",
534+
chatReq(`{"type":"function","function":{"name":""}}`))
535+
536+
Expect(rec.Code).To(Equal(http.StatusOK))
537+
Expect(capturedConfig).ToNot(BeNil())
538+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
539+
})
540+
})
541+
542+
Context("nil tool_choice", func() {
543+
It("is a no-op", func() {
544+
rec := postJSON(app, "/v1/chat/completions",
545+
`{"model":"test-model","messages":[{"role":"user","content":"hi"}]}`)
546+
547+
Expect(rec.Code).To(Equal(http.StatusOK))
548+
Expect(capturedConfig).ToNot(BeNil())
549+
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
550+
Expect(capturedConfig.FunctionToCall()).To(Equal(""))
551+
})
552+
})
553+
})

0 commit comments

Comments
 (0)