Skip to content

Commit 1e7ae16

Browse files
committed
docs(learnings): add scenario-naming convention for sim suites
When the same rubric runs against N personality variants in a suite, give EACH (rubric, personality) pair its own scenario file with a uniquely descriptive name — even if the rubric content is identical. The dashboard displays scenarios by name in run-history; shared-name scenarios across variants make failure investigation painful (every result row looks the same). Recommendation in the new section: - Filename pattern: <base>-<personality-variant>-handling - name field: descriptive, calls out the personality being tested - Rubric content can be identical (cheap mechanical duplication) - Each test (sim) file references its variant-specific scenario Surfaced during PRISM-481 Mudflap iForm sim iteration. Original 4 live-pickup tests shared one scenario; original 6 voicemail-edge-cases tests shared one scenario. Split into per-personality scenarios mid- iteration after the dashboard navigation friction became unworkable. Documenting so future customer suites ship with per-personality scenario naming from day one.
1 parent 605b324 commit 1e7ae16

1 file changed

Lines changed: 45 additions & 0 deletions

File tree

docs/learnings/simulations.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,51 @@ Extra system messages beyond `messages[0]` are **not** included in the tester's
1616

1717
---
1818

19+
## Scenario Naming Convention (CRITICAL for dashboard navigability)
20+
21+
When the same rubric needs to run against multiple personality variants in a sim suite, give EACH `(rubric, personality)` pair its own scenario file with a uniquely descriptive name — even if the rubric content is identical across them.
22+
23+
**Why:** the dashboard's run-history view displays scenarios by `name`, NOT by which personality drove the test. If 4 sims share a scenario named `iForm Live Human Pickup Handling`, all 4 result entries show identically in the suite-run sidebar — you can't tell which test was the "quick" pickup vs the "self-id" pickup vs the "question" pickup vs the "ambiguous-short" pickup without drilling into each item to see the personality. This makes failure investigation painful: every flickering test looks like the same test.
24+
25+
**Recommendation:** name each scenario as `<base>-<personality-variant>-handling`, with a descriptive `name:` field that calls out the personality being tested.
26+
27+
```yaml
28+
# resources/<env>/simulations/scenarios/iform-live-human-pickup-quick-handling.yml
29+
name: iForm Live Human Pickup — Quick (bare hello)
30+
evaluations: [...]
31+
```
32+
33+
```yaml
34+
# resources/<env>/simulations/scenarios/iform-live-human-pickup-self-id-handling.yml
35+
name: iForm Live Human Pickup — Self-ID (driver introduces themselves)
36+
evaluations: [...] # identical rubric content as above; only name differs
37+
```
38+
39+
```yaml
40+
# resources/<env>/simulations/scenarios/iform-live-human-pickup-question-handling.yml
41+
name: iForm Live Human Pickup — Question (skeptical "who's calling?")
42+
evaluations: [...] # same
43+
```
44+
45+
Each test (sim) file then references its variant-specific scenario:
46+
47+
```yaml
48+
# resources/<env>/simulations/tests/iform-live-human-pickup-quick.yml
49+
name: iForm Live Human Pickup - Quick
50+
personalityId: live-human-pickup-quick-bot
51+
scenarioId: iform-live-human-pickup-quick-handling
52+
```
53+
54+
**Cost:** scenario file duplication — each variant is a copy of the same rubric content with a different `name:` field. Cheap. The duplication is mechanical (you can clone the source scenario file 4-6 times with a one-line `name:` change each).
55+
56+
**Pays for itself the first time** you debug a failed run and need to identify which test variant flickered. Particularly important for LLM-as-judge sim suites where the per-evaluator results are subtle and the scenario name is your primary navigation handle.
57+
58+
**Anti-pattern:** putting one shared scenario behind N personality variants in the same suite. The dashboard sidebar shows N rows with identical scenario names, only distinguishable by clicking into each item to see the personality. Sim iteration time inflates because every failure investigation starts with "wait, which one was this?"
59+
60+
Cross-reference: this convention surfaced as friction during the Mudflap iForm Voicemail Triage sim iteration (PRISM-481). Original suites shipped with one shared scenario per group (4 live-pickup tests sharing one scenario, 6 voicemail-edge-cases sharing one scenario); split into per-personality scenarios mid-iteration. Worth shipping new suites in the per-personality form from day one.
61+
62+
---
63+
1964
## Evaluation Comparators
2065

2166
Simulation evaluations support these comparators:

0 commit comments

Comments
 (0)