Skip to content

Commit 9cf6f64

Browse files
committed
docs(learnings): add gpt-5 vs gpt-5-chat-latest reasoning-model gotcha
Documents the platform-level footgun where gpt-5's reasoning tokens silently consume the maxTokens budget, causing tool-only assistants to truncate tool calls and leak text. Recommends gpt-5-chat-latest as the default for tool-only / classifier roles, and gives a 4-5x maxTokens guideline if gpt-5 is intentionally chosen. Surfaced from a real debug on a Mudflap iform-triage-classifier where the symptom set (DTMF firing on humans, reasoning voiced as TTS, handoff never firing) was misdiagnosed as prompt failure before the budget-exhaustion root cause was identified.
1 parent 94e4df4 commit 9cf6f64

1 file changed

Lines changed: 23 additions & 0 deletions

File tree

docs/learnings/assistants.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,29 @@ If omitted, Vapi defaults to `temperature: 0` and `maxTokens: 250`. This can cau
2424

2525
Despite being a configurable field, this setting currently has **no effect** on call behavior. Don't rely on it.
2626

27+
### `gpt-5` is a reasoning model; `gpt-5-chat-latest` is the non-reasoning chat variant
28+
29+
OpenAI's `gpt-5` family on Vapi splits into two distinct shapes:
30+
31+
| Model ID | Behavior | Use it for |
32+
|---|---|---|
33+
| `gpt-5` | **Reasoning model.** Generates internal reasoning tokens before any user-visible output. Reasoning tokens are billed and **count against `maxTokens`**. | Tasks where multi-step deduction over the prompt is the goal. |
34+
| `gpt-5-chat-latest` | **Non-reasoning chat variant.** Behaves like a standard chat completion. No hidden reasoning step. | Conversational SDR, tool-only triage, anything latency-sensitive. |
35+
36+
**The footgun:** `gpt-5`'s reasoning tokens are invisible in the assistant config but are deducted from your `maxTokens` ceiling. A tool-only assistant configured with `model: gpt-5, maxTokens: 60` may have only a handful of tokens left after reasoning to emit a tool call. Symptoms cluster:
37+
38+
- Model emits free-form text instead of the expected tool call ("voicing reasoning out loud") because the tool-call envelope can't fit in the leftover budget.
39+
- When multiple tools are available, the model picks whichever has the cheapest argument shape (e.g. `dtmf` with `keys: "0w"` is ~5 output tokens; a handoff with a sentence-long `reason` is ~25+). This looks like the model is misclassifying the situation but is actually a budget-fit decision.
40+
- The same prompt that worked correctly on `gpt-5-chat-latest` regresses to near-0% pass rate after a swap to `gpt-5` with no other config change.
41+
42+
**Diagnostic signal:** if a model with a clear, well-described tool list still picks the wrong tool *and* the prompt explicitly forbids the picked tool in that case, suspect output-budget exhaustion before you suspect the prompt. Reasoning tokens spent silently bias output toward whatever is shortest.
43+
44+
**Recommendation:**
45+
46+
- **Tool-only / triage / classifier assistants** (must emit a tool call, never TTS): use `gpt-5-chat-latest`. Reasoning capability buys nothing — the job is "match transcript pattern, emit tool call."
47+
- **Conversational assistants where reasoning helps**: stick with `gpt-5-chat-latest` unless you have evidence reasoning is needed. If you do swap to `gpt-5`, set `maxTokens` to **at least 4–5x** the longest plausible visible response to leave headroom for the reasoning step. For a tool-only assistant, that means ≥150 even when the actual tool-call envelope is ~25 tokens.
48+
- **Default for new assistants in this repo:** `gpt-5-chat-latest` matches what every non-classifier assistant uses. Picking `gpt-5` should be a deliberate choice with a documented reason, not the result of grabbing the shortest model name from the API enum.
49+
2750
---
2851

2952
## Voice Configuration

0 commit comments

Comments
 (0)