You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: fern/observability/simulations-quickstart.mdx
+2-72Lines changed: 2 additions & 72 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,15 +35,13 @@ Simulations support two transport modes:
35
35
36
36
<CardGroupcols={2}>
37
37
<Cardtitle="Voice mode"icon="phone">
38
-
**`vapi.websocket`** - Full voice simulation with audio
39
-
38
+
- Full voice simulation with audio
40
39
- Realistic end-to-end testing
41
40
- Tests speech recognition and synthesis
42
41
- Produces call recordings
43
42
</Card>
44
43
<Cardtitle="Chat mode"icon="message">
45
-
**`vapi.webchat`** - Text-based chat simulation
46
-
44
+
- Text-based chat simulation
47
45
- Faster execution
48
46
- Lower cost (no audio processing)
49
47
- Ideal for rapid iteration
@@ -572,17 +570,6 @@ curl -X GET "https://api.vapi.ai/eval/simulation/run/550e8400-e29b-41d4-a716-446
572
570
</Tab>
573
571
</Tabs>
574
572
575
-
### Transport options
576
-
577
-
| Provider | Description | Use Case |
578
-
| --- | --- | --- |
579
-
|`vapi.websocket`| Full voice simulation with audio bridge | Production validation, end-to-end testing |
580
-
|`vapi.webchat`| Text-based chat simulation | Rapid iteration, development testing |
581
-
582
-
<Warning>
583
-
**Hooks are only supported in voice mode.** If your scenario uses hooks (like `call.started` or `call.ended`), you must use `vapi.websocket` transport.
584
-
</Warning>
585
-
586
573
## Step 6: Review results
587
574
588
575
Analyze the results of your simulation runs to understand how your assistant performed.
@@ -690,63 +677,6 @@ curl -X GET "https://api.vapi.ai/eval/simulation/run?status=ended" \
690
677
Full conversation transcripts are available for all simulation runs, making it easy to understand exactly what happened during each test.
691
678
</Note>
692
679
693
-
## Troubleshooting
694
-
695
-
| Issue | Solution |
696
-
| --- | --- |
697
-
| Simulation always fails | Check that evaluations are achievable with your assistant's current capabilities |
698
-
| Run stuck in "running" | Verify your assistant is properly configured and responding |
699
-
| Evaluations not extracting data | Ensure structured output schemas match what your assistant can provide |
700
-
| No audio in recording | Check that you're using `vapi.websocket` transport, not `vapi.webchat`|
701
-
| Hooks not triggering | Hooks only work with `vapi.websocket` transport |
702
-
| Suite won't start | Verify all simulations in the suite exist and have valid scenarios/personalities |
703
-
704
-
### Common errors
705
-
706
-
**"assistant-error" status:**
707
-
708
-
- Check your assistant configuration (model, voice, tools)
709
-
- Verify API keys are valid
710
-
- Test the assistant manually before running simulations
711
-
712
-
**Evaluation failures:**
713
-
714
-
- Review the conversation transcript to understand what happened
715
-
- Adjust structured output schemas to better match expected responses
716
-
- Consider whether the personality is making the test unrealistically difficult
717
-
718
-
<Warning>
719
-
If simulations consistently fail, test your assistant manually first to ensure it's working correctly before debugging the simulation configuration.
720
-
</Warning>
721
-
722
-
## API reference
723
-
724
-
| Endpoint | Method | Description |
725
-
| --- | --- | --- |
726
-
|`/eval/simulation/personality`| POST | Create a personality |
727
-
|`/eval/simulation/personality`| GET | List personalities |
728
-
|`/eval/simulation/personality/:id`| GET | Get a personality |
729
-
|`/eval/simulation/personality/:id`| PATCH | Update a personality |
730
-
|`/eval/simulation/personality/:id`| DELETE | Delete a personality |
731
-
|`/eval/simulation/scenario`| POST | Create a scenario |
732
-
|`/eval/simulation/scenario`| GET | List scenarios |
733
-
|`/eval/simulation/scenario/:id`| GET | Get a scenario |
734
-
|`/eval/simulation/scenario/:id`| PATCH | Update a scenario |
735
-
|`/eval/simulation/scenario/:id`| DELETE | Delete a scenario |
736
-
|`/eval/simulation`| POST | Create a simulation |
737
-
|`/eval/simulation`| GET | List simulations |
738
-
|`/eval/simulation/:id`| GET | Get a simulation |
739
-
|`/eval/simulation/:id`| PATCH | Update a simulation |
740
-
|`/eval/simulation/:id`| DELETE | Delete a simulation |
741
-
|`/eval/simulation/suite`| POST | Create a suite |
742
-
|`/eval/simulation/suite`| GET | List suites |
743
-
|`/eval/simulation/suite/:id`| GET | Get a suite |
744
-
|`/eval/simulation/suite/:id`| PATCH | Update a suite |
745
-
|`/eval/simulation/suite/:id`| DELETE | Delete a suite |
746
-
|`/eval/simulation/run`| POST | Start a simulation run |
747
-
|`/eval/simulation/run`| GET | List runs |
748
-
|`/eval/simulation/run/:id`| GET | Get run details |
0 commit comments