Skip to content

Commit abb294d

Browse files
dhruva-vapiclaude
andcommitted
docs(learnings): idle-message/voicemail conflict, verbose-ban priming, tool-id TTS leaks
Three intertwined gotchas surfaced during a customer voicemail-edge sim hardening pass: - voicemail-detection.md: idleTimeoutSeconds < silenceTimeoutSeconds + a "stay silent on voicemail" prompt strategy deadlocks the moment voicemailDetection has a coverage hole — every failed transcript ends with the idle-message injection. Cannot be fixed at the prompt level. - assistants.md (new "Prompt Authoring" section): verbose negative-directive lists prime the banned phrases under output uncertainty. A 50+ phrase ban regressed a sim suite from 12/15 to 3/15. Short positive directives beat exhaustive negative enumeration. - tools.md: naming a tool resource ID in system-prompt prose causes the LLM to emit it as TTS content instead of (or in addition to) invoking it via tool_calls. Refer to capabilities by natural-language intent. All three entries are generalized — no customer or org-specific identifiers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent da6a176 commit abb294d

3 files changed

Lines changed: 58 additions & 0 deletions

File tree

docs/learnings/assistants.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -532,3 +532,21 @@ When HIPAA is enabled (on org or assistant) and HIPAA provider validation is enf
532532
### `toolIds` and inline `tools` can coexist
533533

534534
They are merged, not mutually exclusive. But be aware of potential duplicates.
535+
536+
---
537+
538+
## Prompt Authoring
539+
540+
### Verbose negative-directive lists prime the banned phrases
541+
542+
Listing many forbidden phrases in a system prompt ("never say 'X', 'Y', 'Z', …") can paradoxically *increase* their emission rate rather than decrease it. Every banned phrase you enumerate is a token plant in the model's active context — under output uncertainty (the rule says "stay silent," but the platform is asking for *some* output), the model preferentially samples from recently-activated tokens. The verbose ban becomes a verbose menu of likely outputs.
543+
544+
**Concrete failure pattern:** In one validation, a 50+ phrase ban list targeting voicemail edge cases regressed a sim-suite pass rate from 80% (12/15) to 20% (3/15). The model emitted nonsense single tokens that mapped to the banned-phrase region of the prompt — short fragments like one-word utterances ending in periods that didn't appear anywhere else in the conversation surface.
545+
546+
**Recommendation:**
547+
- Prefer a short *positive* directive ("emit empty `content`") over an exhaustive negative enumeration.
548+
- If specific phrase bans are necessary, keep the list to 3–5 representative examples and rely on a principle clause ("…or any narration of your intent") rather than exhaustive listing.
549+
- Validate prompt changes against a sim suite before rolling forward — verbose-ban regressions don't show up in single test calls; they require a few iterations of statistical signal to surface.
550+
- This generalizes beyond voicemail: any "stay silent" / "don't say X" rule benefits from positive framing.
551+
552+
Cross-reference: see [voicemail-detection.md](voicemail-detection.md) for the platform-level conflict that often motivates these prompt rules in the first place.

docs/learnings/tools.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,14 @@ If `endCallFunctionEnabled: true` is set on the assistant and no explicit `endCa
123123

124124
When the LLM invokes an endCall tool, the call's `endedReason` is set to `assistant-ended-call` — not `customer-ended-call` or a generic hangup reason. Use this in post-call analysis and structured output evaluations.
125125

126+
### Naming a tool resource ID in system-prompt prose causes TTS leak
127+
128+
If your system prompt references a specific tool by its resource ID ("call the `your-end-call-feature-abc12345` tool"), the model can emit the resource ID as `content` (TTS-bound speech) instead of, or in addition to, invoking it via `tool_calls`. The TTS pipeline speaks the ID aloud as audio. If the call has anything looping audio back through STT (a personality bot in a sim, or a real human listening), the transcript captures partial syllables of the spoken ID.
129+
130+
**Concrete signature:** A tool resource ID like `your-feature-name-abc12345` shows up in the AI transcript as something like `"feature name, a b c one two three four five"` — short, mangled fragments that map character-by-character to the tool ID.
131+
132+
**Recommendation:** Refer to tool capabilities by **natural-language intent** in prose ("end the call", "transfer to a specialist", "look up the customer"), never by resource ID. The tool-calling layer picks the correct registered function from `toolIds` based on intent. If the LLM is reluctant to invoke a tool, fix the tool's `function.description` rather than naming the tool in the prompt body — see [transfers.md](transfers.md) for the same pattern in the transfer-tool context.
133+
126134
---
127135

128136
## handoff Tools

docs/learnings/voicemail-detection.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,3 +269,35 @@ The detection assistant doesn't need post-call analysis — save the LLM call.
269269
| Agent hangs up on humans saying "sorry" | "sorry" misclassified as voicemail context | Add false-positive prevention rules |
270270
| Agent stuck on IVR forever | No timeout or exit condition | Set `silenceTimeoutSeconds` and `maxDurationSeconds` |
271271
| Greeting plays twice (handoff + fronter firstMessage) | Fronter has its own `firstMessage` | Set fronter `firstMessage: ""` or use model-generated mode |
272+
| Idle message ("Are you still there?") fires on voicemail edge cases | `idleTimeoutSeconds` < `silenceTimeoutSeconds` while assistant strategy is "stay silent on voicemail" | See "Idle messages collide with silence-based voicemail handling" below |
273+
274+
---
275+
276+
## Idle Messages and Voicemail Silence
277+
278+
### Idle messages collide with silence-based voicemail handling
279+
280+
When `messagePlan.idleMessages` is set with a tight `idleTimeoutSeconds` (e.g. 6s) AND the voicemail strategy is "stay silent and let platform-level `voicemailDetection` hang up the call," the two settings deadlock on every edge case where Vapi's detection has a coverage hole (full-mailbox prompts, some carrier-specific custom greetings, late-beep messages).
281+
282+
**What you might expect:** The static `voicemailMessage` plays and the call ends. Idle messages don't fire because there's no assistant-side silence — the platform handles everything.
283+
284+
**What actually happens:** When `voicemailDetection` misses, the LLM gets a turn. If the system prompt instructs it to stay silent on voicemail signals, the assistant goes silent — and `idleMessages` injects "Are you still there?" after `idleTimeoutSeconds`. Result: every voicemail-edge call ends with the idle prompt in the transcript, breaking any sim rubric that scores `idle_prompt_after_voicemail` or any post-call analysis that flags assistant utterances on voicemail. Cannot be fixed at the prompt level — even a perfect-silence prompt can't beat the platform timer.
285+
286+
**Concrete signature:** Every failed transcript on a voicemail-edge sim suite ends with `AI: Are you still there?` while `endedReason` is `silence-timed-out`.
287+
288+
**Recommendation:**
289+
- **Preferred — two-agent relay.** Use a classifier squad member that ends the call before the main assistant ever gets a voicemail turn. The main assistant's `idleTimeoutSeconds` stays tight for human conversations.
290+
- **Fallback — single-agent.** Set `idleTimeoutSeconds` >= `silenceTimeoutSeconds` so silence-timeout terminates the call before idle injection. This costs you idle-prompt responsiveness for genuinely-silent humans but eliminates the conflict.
291+
- Don't try to suppress this with system-prompt rules — the idle injection happens at the platform layer, not the LLM layer.
292+
293+
```yaml
294+
# Single-agent fallback (less ideal — loses tight human-side idle responsiveness)
295+
silenceTimeoutSeconds: 15
296+
messagePlan:
297+
idleMessageMaxSpokenCount: 1
298+
idleMessages:
299+
- Are you still there?
300+
idleTimeoutSeconds: 30 # >= silenceTimeoutSeconds → idle never fires before call ends
301+
```
302+
303+
Cross-reference: see [squads.md](squads.md) for the two-agent relay pattern, and [assistants.md](assistants.md) for prompt-authoring guidance on silence rules.

0 commit comments

Comments
 (0)