You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(learnings): add gpt-5 vs gpt-5-chat-latest reasoning-model gotcha
Documents the platform-level footgun where gpt-5's reasoning tokens
silently consume the maxTokens budget, causing tool-only assistants
to truncate tool calls and leak text. Recommends gpt-5-chat-latest
as the default for tool-only / classifier roles, and gives a 4-5x
maxTokens guideline if gpt-5 is intentionally chosen.
Surfaced from a real debug on a Mudflap iform-triage-classifier where
the symptom set (DTMF firing on humans, reasoning voiced as TTS,
handoff never firing) was misdiagnosed as prompt failure before the
budget-exhaustion root cause was identified.
Copy file name to clipboardExpand all lines: docs/learnings/assistants.md
+23Lines changed: 23 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,6 +24,29 @@ If omitted, Vapi defaults to `temperature: 0` and `maxTokens: 250`. This can cau
24
24
25
25
Despite being a configurable field, this setting currently has **no effect** on call behavior. Don't rely on it.
26
26
27
+
### `gpt-5` is a reasoning model; `gpt-5-chat-latest` is the non-reasoning chat variant
28
+
29
+
OpenAI's `gpt-5` family on Vapi splits into two distinct shapes:
30
+
31
+
| Model ID | Behavior | Use it for |
32
+
|---|---|---|
33
+
|`gpt-5`|**Reasoning model.** Generates internal reasoning tokens before any user-visible output. Reasoning tokens are billed and **count against `maxTokens`**. | Tasks where multi-step deduction over the prompt is the goal. |
34
+
|`gpt-5-chat-latest`|**Non-reasoning chat variant.** Behaves like a standard chat completion. No hidden reasoning step. | Conversational SDR, tool-only triage, anything latency-sensitive. |
35
+
36
+
**The footgun:**`gpt-5`'s reasoning tokens are invisible in the assistant config but are deducted from your `maxTokens` ceiling. A tool-only assistant configured with `model: gpt-5, maxTokens: 60` may have only a handful of tokens left after reasoning to emit a tool call. Symptoms cluster:
37
+
38
+
- Model emits free-form text instead of the expected tool call ("voicing reasoning out loud") because the tool-call envelope can't fit in the leftover budget.
39
+
- When multiple tools are available, the model picks whichever has the cheapest argument shape (e.g. `dtmf` with `keys: "0w"` is ~5 output tokens; a handoff with a sentence-long `reason` is ~25+). This looks like the model is misclassifying the situation but is actually a budget-fit decision.
40
+
- The same prompt that worked correctly on `gpt-5-chat-latest` regresses to near-0% pass rate after a swap to `gpt-5` with no other config change.
41
+
42
+
**Diagnostic signal:** if a model with a clear, well-described tool list still picks the wrong tool *and* the prompt explicitly forbids the picked tool in that case, suspect output-budget exhaustion before you suspect the prompt. Reasoning tokens spent silently bias output toward whatever is shortest.
43
+
44
+
**Recommendation:**
45
+
46
+
-**Tool-only / triage / classifier assistants** (must emit a tool call, never TTS): use `gpt-5-chat-latest`. Reasoning capability buys nothing — the job is "match transcript pattern, emit tool call."
47
+
-**Conversational assistants where reasoning helps**: stick with `gpt-5-chat-latest` unless you have evidence reasoning is needed. If you do swap to `gpt-5`, set `maxTokens` to **at least 4–5x** the longest plausible visible response to leave headroom for the reasoning step. For a tool-only assistant, that means ≥150 even when the actual tool-call envelope is ~25 tokens.
48
+
-**Default for new assistants in this repo:**`gpt-5-chat-latest` matches what every non-classifier assistant uses. Picking `gpt-5` should be a deliberate choice with a documented reason, not the result of grabbing the shortest model name from the API enum.
0 commit comments