docs: add learnings from B2B pilot — pronunciation, keyterm, simulations, squad patterns

dhruva-vapi · dhruva-vapi · commit 8e8a4e925bc5 · 2026-04-15T16:41:39.000-07:00
- assistants.md: Deepgram Nova-3 keyterm vs keywords, pronunciation dictionary provider comparison (Cartesia/ElevenLabs/Vapi), three-layer pronunciation approach
- tools.md: dead air during KB/API tool calls — request-start + request-response-delayed fix pattern
- squads.md: toolIds in assistantOverrides require UUIDs (not filenames), FAQ agent consolidation pattern
- simulations.md: running simulations against squads, A/B testing workflow, primitive-type evaluation constraint, filename renaming after push
- multilingual.md: updated best single-agent stack to Cartesia sonic-3, added keyterm example
- latency.md: added Cartesia pronunciation row to TTS selection table
diff --git a/docs/learnings/assistants.md b/docs/learnings/assistants.md
@@ -62,8 +62,48 @@ If omitted, transcripts with confidence below 0.4 may be **ignored** entirely (n
 
 [Smart Denoising (Krisp)](https://docs.vapi.ai/documentation/assistants/conversation-behavior/background-speech-denoising#smart-denoising-krisp) is recommended over [Fourier Denoising](https://docs.vapi.ai/documentation/assistants/conversation-behavior/background-speech-denoising#fourier-denoising-experimental) (experimental). Enable it via `backgroundDenoisingEnabled: true` or `smartDenoisingPlan.enabled: true`.
 
+### Custom keyword/keyterm boosting
+
 Boost domain-specific vocabulary with [Custom Keywords](https://docs.vapi.ai/customization/custom-keywords) to improve recognition of brand names, product names, and industry terms.
 
+**Nova-3 uses `keyterm` (not `keywords`).** The legacy `keywords` field only works on Nova-2 and older models. For Nova-3, use `keyterm` — an array of words or multi-word phrases (no intensifiers). Supports up to 100 terms (~500 tokens).
+
+**`keyterm` works in multilingual mode.** As of November 2025, `model: nova-3` with `language: multi` supports keyterm prompting. Previously this combination returned a 400 error.
+
+```yaml
+transcriber:
+  provider: deepgram
+  model: nova-3
+  language: multi
+  keyterm:
+    - your-brand-name
+    - industry-specific-term
+    - product-name
+    - technical-acronym
+```
+
+### Pronunciation dictionaries (TTS-level)
+
+Pronunciation dictionaries control how TTS voices say specific words. They are **provider-specific**:
+
+| Provider | Support | Config field | Model requirement |
+|----------|---------|-------------|-------------------|
+| **Cartesia** | Full IPA + sounds-like across all languages | `pronunciationDictId` on voice config | `sonic-3` only |
+| **ElevenLabs** | Phoneme rules (IPA/CMU, English only) + alias rules (all languages) | `pronunciationDictionaryLocators` on voice config | Phoneme: `eleven_turbo_v2`, `eleven_flash_v2`. Alias: all models |
+| **Vapi built-in** | None | N/A | N/A |
+
+**Cartesia pronunciation dictionaries** are created via the Vapi API (`POST /provider/cartesia/pronunciation-dictionary`), then referenced by ID in the voice config. This is the same pattern as `credentialId` — the provider resource lives outside gitops, the reference is gitops-managed.
+
+```yaml
+voice:
+  provider: cartesia
+  model: sonic-3
+  voiceId: your-voice-id
+  pronunciationDictId: pdict_xxxxxxxxxxxxx
+```
+
+**Recommendation:** For multilingual use cases, Cartesia sonic-3 with a pronunciation dictionary is the strongest option — IPA works across all languages. Combine with prompt-level pronunciation rules (belt-and-suspenders) and transcriber `keyterm` for a three-layer approach: TTS output, LLM text generation, and STT input.
+
 ### `smartEndpointingPlan` **owns** turn detection when set
 
 If you configure `startSpeakingPlan.smartEndpointingPlan`, the transcriber's own `endpointing` settings (voice activity detection timeouts, etc.) are **not used** for turn detection. Smart endpointing takes full control.
diff --git a/docs/learnings/latency.md b/docs/learnings/latency.md
@@ -122,7 +122,10 @@ See [assistants.md](assistants.md) for `stopSpeakingPlan` defaults.
 
 | Priority | Provider type | Examples |
 |----------|--------------|---------|
-| Lowest latency | Low-latency conversational voices | Vapi built-in, Cartesia Sonic, Deepgram Aura |
+| Lowest latency | Low-latency conversational voices | Cartesia Sonic-3, Vapi built-in, Deepgram Aura |
 | Best quality | High-fidelity voices | ElevenLabs Multilingual v2, PlayHT |
+| Best multilingual + pronunciation control | IPA dictionaries across all languages | Cartesia Sonic-3 with `pronunciationDictId` |
 
 For conversation, **responsiveness almost always wins over voice quality**. A slight quality reduction that saves 100ms of Time to First Audio Byte is worth it in most use cases.
+
+**Cartesia Sonic-3** is Vapi's default voice provider — sub-200ms latency across 42 languages with pronunciation dictionary support. For multilingual use cases needing pronunciation control, it's the strongest choice.
diff --git a/docs/learnings/multilingual.md b/docs/learnings/multilingual.md
@@ -211,17 +211,23 @@ transcriber:
   provider: deepgram
   model: nova-3
   language: multi
+  keyterm:    # Nova-3 multilingual supports keyterm since Nov 2025
+    - your-brand-name
+    - domain-specific-term
 
 voice:
-  provider: eleven-labs
-  model: eleven_multilingual_v2
+  provider: cartesia
+  model: sonic-3
   voiceId: your-voice-id
+  pronunciationDictId: pdict_xxx  # IPA works across all languages
 
 model:
   provider: openai
   model: gpt-4.1
 ```
 
+**Alternative voice:** ElevenLabs `eleven_multilingual_v2` if you need ElevenLabs-specific features (alias rules work across all languages, but IPA phoneme rules are English-only).
+
 ### Best Two-Agent Stack
 
 ```yaml
@@ -240,7 +246,7 @@ voice: { provider: eleven-labs, voiceId: your-spanish-voice }
 
 | Pitfall | Root Cause | Solution |
 |---------|-----------|----------|
-| Agent understands Spanish but speaks English | TTS voice is English-only | Use multilingual TTS (ElevenLabs multilingual_v2, OpenAI) |
+| Agent understands Spanish but speaks English | TTS voice is English-only | Use multilingual TTS (Cartesia sonic-3, ElevenLabs multilingual_v2, OpenAI) |
 | Tool messages always in English | Active language defaults to `"en"` with `"multi"` STT | Use `contents[]` with explicit language variants |
 | Spanish STT accuracy worse than dedicated | Multi-language models trade accuracy for flexibility | Use dedicated per-language assistants (Approach 2) |
 | Self-handoff infinite loop | LLM re-triggers handoff after seeing same conversation | Clear prompt: "Do not trigger language switch if already in correct language" |
diff --git a/docs/learnings/simulations.md b/docs/learnings/simulations.md
@@ -50,3 +50,47 @@ Evaluations that require audio (`target: messages-with-audio`) are **skipped** i
 - **At run creation (API):** Missing scenario or personality IDs fail the API request immediately with a validation error.
 - **At execution:** Missing references fail the run. Retries depend on platform configuration.
 - **Inline runs:** Require snapshots in metadata or fail with a "no inline config" error.
+
+---
+
+## Running Simulations Against Squads
+
+Simulations can target squads directly using `target.type: "squad"` with the squad ID:
+
+```bash
+curl -X POST "https://api.vapi.ai/eval/simulation/suite/{suiteId}/run" \
+  -H "Authorization: Bearer $VAPI_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"target": {"type": "squad", "squadId": "your-squad-id"}, "transport": {"provider": "vapi.websocket"}, "iterations": 3}'
+```
+
+**Voice mode (`vapi.websocket`)** exercises the full STT/TTS pipeline — use for latency testing and realistic end-to-end validation. **Chat mode (`vapi.webchat`)** is faster and cheaper — use for rapid iteration on outcome evaluations.
+
+**Squad simulations test the full stack**: all tools attached to squad members (KB lookups, logging, email tools) actually fire during the simulation. If a tool endpoint is down, the simulation produces different results. Factor this into test design.
+
+---
+
+## A/B Testing Squads with Simulations
+
+Run the same simulation suite against two squad variants and compare results:
+
+1. Create both squad variants as separate resources (e.g., Squad A vs Squad B)
+2. Run the same suite against each with identical iterations
+3. Compare pass/fail rates on structured output evaluations
+4. For latency comparison, use voice mode and analyze `end-of-call-report` webhook data from each run
+
+**Limitation:** Simulations evaluate outcomes via structured output comparators, not platform-level metrics. There is no built-in p95 latency or dead-air measurement. For quantitative latency data, pipe `end-of-call-report` webhooks to your own analytics and compute metrics from `artifact.performanceMetrics`.
+
+---
+
+## Structured Outputs in Simulation Evaluations
+
+Scenario evaluations reference structured outputs either by `structuredOutputId` (existing resource) or inline via `structuredOutput`. The key constraint: **evaluation schemas must use primitive types** (`string`, `number`, `integer`, `boolean`).
+
+If your post-call analytics structured output uses `type: object` with nested schemas, you **cannot** use it directly as a scenario evaluation. Instead, create separate primitive-typed structured outputs for each evaluation criterion (e.g., `eval-call-outcome` as `type: string`, `eval-goal-achieved` as `type: boolean`). The full analytics output can still run via the squad's `artifactPlan.structuredOutputIds` — it just can't be used with comparators.
+
+---
+
+## Simulation File Names After Push
+
+Simulation resource files use placeholder UUIDs (`a0000000`) locally. After the first push, the gitops engine creates platform resources and maps local filenames to platform UUIDs in `.vapi-state.<env>.json`. On subsequent state syncs (bootstrap), filenames may be updated to include the platform name — this triggers `name_mismatch` warnings that are resolved automatically by re-running bootstrap.
diff --git a/docs/learnings/squads.md b/docs/learnings/squads.md
@@ -99,3 +99,33 @@ The final assistant configuration is built by merging these layers in order:
 Later layers win on conflicts. `variableValues` from all layers are merged separately.
 
 **Gotcha:** Liquid template substitution replaces undefined variables with **empty strings**, not errors. If a variable isn't in the merged `variableValues`, any `{{ myVar }}` reference silently becomes `""`.
+
+---
+
+## toolIds in assistantOverrides require UUIDs
+
+`assistantOverrides.model.toolIds` in squad members must use **Vapi platform UUIDs**, not local filenames. The gitops engine resolves filenames to UUIDs for base assistant `model.toolIds`, but it does **not** resolve them inside squad `assistantOverrides`. If you use a local filename, the push will fail with `each value in toolIds must be a UUID`.
+
+**Workaround:** Look up the tool's UUID from `.vapi-state.<env>.json` and use it directly.
+
+```yaml
+# WRONG — filename, will fail in assistantOverrides
+assistantOverrides:
+  model:
+    toolIds:
+      - my-tool-name
+
+# CORRECT — platform UUID
+assistantOverrides:
+  model:
+    toolIds:
+      - a1b2c3d4-e5f6-7890-abcd-ef1234567890
+```
+
+---
+
+## FAQ agent consolidation pattern
+
+When a squad has multiple specialist agents that each carry one knowledge base tool, the LLM must correctly classify and route the question before it even reaches a KB. If the routing is wrong, the KB returns "I don't have enough information" — not because the knowledge doesn't exist, but because the wrong KB was queried.
+
+**Fix:** Consolidate specialist agents into a single FAQ agent with access to all KB tools. The FAQ agent's LLM picks the right tool based on improved tool descriptions with explicit routing boundaries and "Do NOT use for..." cross-references. This eliminates the routing classification step from the handoff layer and moves it to the tool selection layer, where descriptions give the LLM more direct guidance.
diff --git a/docs/learnings/tools.md b/docs/learnings/tools.md
@@ -199,6 +199,27 @@ An unrecognized message type is coerced to `request-failed` with default error c
 
 After one delayed message plays, another won't play for at least 2.5 seconds — even if multiple timing thresholds are crossed. Missing `timingMilliseconds` defaults to the 0ms bucket.
 
+### Dead air during KB/API tool calls
+
+If a tool has no `request-start` content (or empty content), the caller hears silence while the tool executes. For knowledge base tools and API requests that take 2–5 seconds, this feels like dead air.
+
+**Fix with two layers:**
+
+1. **`request-start`** with `blocking: false` — speaks a filler line ("Good question — let me look that up") in parallel with the tool call starting.
+2. **`request-response-delayed`** at 4000ms — safety net if the tool takes longer than expected.
+
+```yaml
+messages:
+  - type: request-start
+    content: "Good question — let me look that up."
+    blocking: false
+  - type: request-response-delayed
+    content: "Still looking that up for you."
+    timingMilliseconds: 4000
+```
+
+Optionally add prompt-level instructions ("say a brief acknowledgment before calling the tool") as a belt-and-suspenders approach — the prompt handles cases where the LLM speaks before calling the tool, while the tool message handles cases where the LLM calls the tool silently.
+
 ---
 
 ## voicemail Tools
