/v1/responses silently returns empty output: convertORInputToMessages leaves Content nil; array items without type:"message" are dropped

[rsclafani]

LocalAI version:

master @ 373dc449923e (image localai/localai:sha-373dc44-nvidia-l4t-arm64-cuda-13, pushed 2026-05-27 20:48 UTC).

Environment, CPU architecture, OS, and Version:

NVIDIA DGX Spark — GB10 GPU, 128 GB unified memory, 12-core Grace ARM64 CPU. Ubuntu 24.04 LTS arm64 host. Backend: cuda13-nvidia-l4t-arm64-llama-cpp.

$ uname -a
Linux <host> 6.17.0-1018-nvidia #18-Ubuntu SMP PREEMPT_DYNAMIC aarch64 aarch64 aarch64 GNU/Linux

Describe the bug

/v1/responses (Open Responses) silently produces empty output for any model whose YAML doesn't include a template.chat_message Go-side template block. The handler builds an empty prompt before templating, sends it to the backend, hits the 5× empty-retry guard in core/http/endpoints/openai/inference.go:129, and returns a response.completed event with output: []. HTTP status is 200 so there's no client-visible error.

/v1/chat/completions against the same model works correctly — its request flows through middleware that populates Message.Content, then llama.cpp's chat endpoint applies the embedded jinja template server-side.

Two cooperating bugs in core/http/endpoints/openresponses/responses.go + core/templates/evaluator.go, plus a third related silent-drop:

Bug A — convertORInputToMessages leaves Content nil for string input

For input: "...", convertORInputToMessages returns (line 302):

return []schema.Message{{Role: "user", StringContent: v}}, nil

It populates StringContent but never sets Content (the any field).

Bug B — TemplateMessages requires both fields to render content

core/templates/evaluator.go:114:

contentExists := i.Content != nil && i.StringContent != ""

With Content == nil, contentExists is false. The fallback content blocks (around lines 168–195) are all gated on contentExists and never execute. mess accumulates "", strings.Join returns "", the backend receives an empty prompt.

Bug C (related) — array input without type:"message" is silently dropped

For input: [{"role":"user","content":"..."}], the array branch of convertORInputToMessages inspects itemMap["type"] and only handles "message", "reasoning", "function_call", "function_call_output", "item_reference". A bare {role, content} object has no type field, falls through the switch with no default, and is dropped. Same downstream effect.

A question for maintainers: should LocalAI accept the bare {role, content} shape here? The openai-python SDK's client.responses.create(input=[{"role":"user","content":"..."}]) sends items without a type discriminator, and LiteLLM's hosted_vllm/ bridge passes them through unchanged — so if the intent is OpenAI-SDK compatibility, dropping these silently is probably not the desired behavior. Happy to be corrected if the spec interpretation here is stricter than I'm reading it.

To Reproduce

1. Start LocalAI master with a llama-cpp model whose YAML does not include a template.chat_message Go-side block. Reproduces with any GGUF; example YAML used here:

   name: qwen3-coder-30b-a3b-test
   backend: cuda13-nvidia-l4t-arm64-llama-cpp
   mmap: false
   parameters:
     model: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
   context_size: 65536
   f16: true
   gpu_layers: 999
   use_tokenizer_template: true
   use_jinja: true
   template:
     chat: |
       {{.Input -}}
   stopwords:
     - '<|im_end|>'
     - '<|endoftext|>'
     - '</tool_call>'

2. Send a minimal /v1/responses request with a plain string input:

   curl -sN -X POST http://localhost:8080/v1/responses \
     -H "Content-Type: application/json" \
     -d '{
       "model": "qwen3-coder-30b-a3b-test",
       "stream": true,
       "input": "Say hello in one word."
     }'

3. Observe the SSE response: response.created → response.in_progress → empty output_item.added/output_item.done → response.completed with "output":[] and "usage":{"output_tokens":0}. No output_text.delta events are emitted.

4. The array-input variant reproduces Bug C:

   curl -sN -X POST http://localhost:8080/v1/responses \
     -H "Content-Type: application/json" \
     -d '{
       "model": "qwen3-coder-30b-a3b-test",
       "stream": true,
       "input": [
         {"role":"user","content":"Say hello in one word."}
       ]
     }'

5. Confirm chat-completions works against the same model:

   curl -sN -X POST http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"qwen3-coder-30b-a3b-test","stream":true,"messages":[{"role":"user","content":"Say hello in one word."}]}'

Expected behavior

• /v1/responses with input: "string" produces a non-empty output_text.delta stream and a non-empty output array on response.completed.
• /v1/responses with input: [{"role":"user","content":"…"}] is handled identically — either accepted, or rejected with a clear 4xx error explaining the missing type field.
• Behavior matches /v1/chat/completions against the same model.

Logs

LocalAI server logs (DEBUG=true) for one repro call:

DEBUG Prompt (before templating) prompt=""           caller=core/templates/evaluator.go:206
DEBUG Template found, input modified input=""        caller=core/templates/evaluator.go:224
DEBUG Open Responses - Prompt (after templating) prompt=""   caller=core/http/endpoints/openresponses/responses.go:233
DEBUG GRPC stderr line="W slot update_slots: id 0 | task N | empty prompt - releasing slot"
DEBUG GRPC stderr line="I slot print_timing: id 0 | task N | prompt eval time = 0.00 ms / 0 tokens"
WARN  Backend returned empty response, retrying attempt=1 maxRetries=5
WARN  Backend returned empty response, retrying attempt=2 maxRetries=5
WARN  Backend returned empty response, retrying attempt=3 maxRetries=5
WARN  Backend returned empty response, retrying attempt=4 maxRetries=5
WARN  Backend returned empty response, retrying attempt=5 maxRetries=5
DEBUG [ChatDeltas] OpenResponses Stream: no pre-parsed tool calls, falling back to Go-side text parsing
DEBUG Open Responses Stream - Parsed toolCalls=0 textContent=""
DEBUG Stored Open Responses response items_count=0
INFO  HTTP request method=POST path=/v1/responses status=200

Client-side SSE (string-input case):

event: response.created
event: response.in_progress
event: response.completed
data: {"type":"response.completed", "response":{"output":[], "usage":{"input_tokens":0,"output_tokens":0,"total_tokens":0}, ...}}
data: [DONE]

Additional context

Solutions attempted

Tried these against the same model YAML before isolating the root cause; recording so others can save the time:

• use_tokenizer_template: true + use_jinja: true (per the post-Regression: Reasoning/thinking output provided as regular output #9985 gallery shape used in PR fix(streaming/tools): don't leak prefill-misclassified content as trailing reasoning chunk #10000) — no change. The handler still logs Prompt (before templating) prompt="" because the message-array passed to TemplateMessages has Content == nil, which is upstream of any templating-engine choice.
• Explicit template.chat: "{{.Input -}}" — no change. The template runs fine, but predInput is already "" when it reaches the template engine.
• input as plain string vs input as [{role,content}] array — both empty (Bug A and Bug C respectively).
• Tools omitted entirely — empty prompt still fires, confirming this is independent of function-calling configuration.

Suspected fix shape

convertORInputToMessages should set both Content and StringContent so that downstream consumers (TemplateMessages, reasoning extraction, etc.) see a consistent message:

case string:
    return []schema.Message{{
        Role:          "user",
        Content:       v,        // <-- add this
        StringContent: v,
    }}, nil

The same fix is needed in convertORMessageItem and any other site that builds schema.Message from openresponses request shapes.

For Bug C, the array-input switch should accept message-shaped items without an explicit type field (treat as type: "message" by default) or return a 4xx with a clear error message, rather than silently dropping them.

CI gap

mock-model-mcp in tests/e2e/e2e_mcp_test.go doesn't catch this for two reasons: it has a Go-side template.chat block, and its mock backend doesn't care about an empty prompt. A regression test against a real model fixture (or a mock that requires a non-empty prompt) would have caught it.

Out of scope

• The use_tokenizer_template: true semantics for openresponses — the fix here is about the conversion step before any templating runs, not about which template engine takes over downstream.
• The 5× empty-retry guard in inference.go:129 — it's correctly firing; the upstream cause is an empty prompt, not a backend issue.

Related history

Pointers for context, not duplicates (searched both open and closed issues/PRs; no existing report names this code path or the empty-prompt symptom):

• feat(api): add support for open responses specification #8063 — original Open Responses feature PR; introduced convertORInputToMessages with the current string/array branches.
• chore: refactor endpoints to use same inferencing path, add automatic retrial mechanism in case of errors #9029 — refactor unifying the openresponses inference path through ComputeChoice plus the retry mechanism. The 5× retry on empty backend responses is what turns Bug A/B from a visible error into a silent empty completion.
• fix: merge openresponses messages #8615 — earlier openresponses message-merging fix (different layer; not a duplicate).