Skip to content

Commit 3690422

Browse files
author
Noah Raynor
committed
a few fixes
1 parent 2a47354 commit 3690422

2 files changed

Lines changed: 2 additions & 97 deletions

File tree

fern/observability/simulations-advanced.mdx

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -544,31 +544,6 @@ Test boundary conditions and unusual inputs your assistant might encounter.
544544
| Pre-production validation | Voice (`vapi.websocket`) | Full end-to-end |
545545
| Testing hooks/webhooks | Voice (`vapi.websocket`) | Hooks require voice |
546546

547-
### Performance optimization
548-
549-
**Minimize test execution time:**
550-
551-
1. **Use chat mode for initial development:**
552-
- `vapi.webchat` runs faster than voice
553-
- Switch to voice for final validation
554-
555-
2. **Keep instructions focused:**
556-
- Clear, specific instructions complete faster
557-
- Avoid overly complex multi-step scenarios
558-
559-
3. **Batch related tests:**
560-
- Group similar simulations in suites
561-
- Run suites instead of individual simulations
562-
563-
4. **Use iterations strategically:**
564-
- `iterations: 1` for deterministic scenarios
565-
- Higher iterations for testing consistency
566-
567-
**Cost optimization:**
568-
- Chat mode costs less (no STT/TTS)
569-
- Simple personalities (smaller models) cost less
570-
- Shorter conversations cost less
571-
572547
## CI/CD integration
573548

574549
Automate simulation runs in your deployment pipeline.

fern/observability/simulations-quickstart.mdx

Lines changed: 2 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,13 @@ Simulations support two transport modes:
3535

3636
<CardGroup cols={2}>
3737
<Card title="Voice mode" icon="phone">
38-
**`vapi.websocket`** - Full voice simulation with audio
39-
38+
- Full voice simulation with audio
4039
- Realistic end-to-end testing
4140
- Tests speech recognition and synthesis
4241
- Produces call recordings
4342
</Card>
4443
<Card title="Chat mode" icon="message">
45-
**`vapi.webchat`** - Text-based chat simulation
46-
44+
- Text-based chat simulation
4745
- Faster execution
4846
- Lower cost (no audio processing)
4947
- Ideal for rapid iteration
@@ -572,17 +570,6 @@ curl -X GET "https://api.vapi.ai/eval/simulation/run/550e8400-e29b-41d4-a716-446
572570
</Tab>
573571
</Tabs>
574572

575-
### Transport options
576-
577-
| Provider | Description | Use Case |
578-
| --- | --- | --- |
579-
| `vapi.websocket` | Full voice simulation with audio bridge | Production validation, end-to-end testing |
580-
| `vapi.webchat` | Text-based chat simulation | Rapid iteration, development testing |
581-
582-
<Warning>
583-
**Hooks are only supported in voice mode.** If your scenario uses hooks (like `call.started` or `call.ended`), you must use `vapi.websocket` transport.
584-
</Warning>
585-
586573
## Step 6: Review results
587574

588575
Analyze the results of your simulation runs to understand how your assistant performed.
@@ -690,63 +677,6 @@ curl -X GET "https://api.vapi.ai/eval/simulation/run?status=ended" \
690677
Full conversation transcripts are available for all simulation runs, making it easy to understand exactly what happened during each test.
691678
</Note>
692679

693-
## Troubleshooting
694-
695-
| Issue | Solution |
696-
| --- | --- |
697-
| Simulation always fails | Check that evaluations are achievable with your assistant's current capabilities |
698-
| Run stuck in "running" | Verify your assistant is properly configured and responding |
699-
| Evaluations not extracting data | Ensure structured output schemas match what your assistant can provide |
700-
| No audio in recording | Check that you're using `vapi.websocket` transport, not `vapi.webchat` |
701-
| Hooks not triggering | Hooks only work with `vapi.websocket` transport |
702-
| Suite won't start | Verify all simulations in the suite exist and have valid scenarios/personalities |
703-
704-
### Common errors
705-
706-
**"assistant-error" status:**
707-
708-
- Check your assistant configuration (model, voice, tools)
709-
- Verify API keys are valid
710-
- Test the assistant manually before running simulations
711-
712-
**Evaluation failures:**
713-
714-
- Review the conversation transcript to understand what happened
715-
- Adjust structured output schemas to better match expected responses
716-
- Consider whether the personality is making the test unrealistically difficult
717-
718-
<Warning>
719-
If simulations consistently fail, test your assistant manually first to ensure it's working correctly before debugging the simulation configuration.
720-
</Warning>
721-
722-
## API reference
723-
724-
| Endpoint | Method | Description |
725-
| --- | --- | --- |
726-
| `/eval/simulation/personality` | POST | Create a personality |
727-
| `/eval/simulation/personality` | GET | List personalities |
728-
| `/eval/simulation/personality/:id` | GET | Get a personality |
729-
| `/eval/simulation/personality/:id` | PATCH | Update a personality |
730-
| `/eval/simulation/personality/:id` | DELETE | Delete a personality |
731-
| `/eval/simulation/scenario` | POST | Create a scenario |
732-
| `/eval/simulation/scenario` | GET | List scenarios |
733-
| `/eval/simulation/scenario/:id` | GET | Get a scenario |
734-
| `/eval/simulation/scenario/:id` | PATCH | Update a scenario |
735-
| `/eval/simulation/scenario/:id` | DELETE | Delete a scenario |
736-
| `/eval/simulation` | POST | Create a simulation |
737-
| `/eval/simulation` | GET | List simulations |
738-
| `/eval/simulation/:id` | GET | Get a simulation |
739-
| `/eval/simulation/:id` | PATCH | Update a simulation |
740-
| `/eval/simulation/:id` | DELETE | Delete a simulation |
741-
| `/eval/simulation/suite` | POST | Create a suite |
742-
| `/eval/simulation/suite` | GET | List suites |
743-
| `/eval/simulation/suite/:id` | GET | Get a suite |
744-
| `/eval/simulation/suite/:id` | PATCH | Update a suite |
745-
| `/eval/simulation/suite/:id` | DELETE | Delete a suite |
746-
| `/eval/simulation/run` | POST | Start a simulation run |
747-
| `/eval/simulation/run` | GET | List runs |
748-
| `/eval/simulation/run/:id` | GET | Get run details |
749-
750680
## Next steps
751681

752682
<CardGroup cols={2}>

0 commit comments

Comments
 (0)