You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(learnings): add scenario-naming convention for sim suites
When the same rubric runs against N personality variants in a suite, give
EACH (rubric, personality) pair its own scenario file with a uniquely
descriptive name — even if the rubric content is identical. The dashboard
displays scenarios by name in run-history; shared-name scenarios across
variants make failure investigation painful (every result row looks the
same).
Recommendation in the new section:
- Filename pattern: <base>-<personality-variant>-handling
- name field: descriptive, calls out the personality being tested
- Rubric content can be identical (cheap mechanical duplication)
- Each test (sim) file references its variant-specific scenario
Surfaced during PRISM-481 Mudflap iForm sim iteration. Original 4
live-pickup tests shared one scenario; original 6 voicemail-edge-cases
tests shared one scenario. Split into per-personality scenarios mid-
iteration after the dashboard navigation friction became unworkable.
Documenting so future customer suites ship with per-personality scenario
naming from day one.
Copy file name to clipboardExpand all lines: docs/learnings/simulations.md
+45Lines changed: 45 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,6 +16,51 @@ Extra system messages beyond `messages[0]` are **not** included in the tester's
16
16
17
17
---
18
18
19
+
## Scenario Naming Convention (CRITICAL for dashboard navigability)
20
+
21
+
When the same rubric needs to run against multiple personality variants in a sim suite, give EACH `(rubric, personality)` pair its own scenario file with a uniquely descriptive name — even if the rubric content is identical across them.
22
+
23
+
**Why:** the dashboard's run-history view displays scenarios by `name`, NOT by which personality drove the test. If 4 sims share a scenario named `iForm Live Human Pickup Handling`, all 4 result entries show identically in the suite-run sidebar — you can't tell which test was the "quick" pickup vs the "self-id" pickup vs the "question" pickup vs the "ambiguous-short" pickup without drilling into each item to see the personality. This makes failure investigation painful: every flickering test looks like the same test.
24
+
25
+
**Recommendation:** name each scenario as `<base>-<personality-variant>-handling`, with a descriptive `name:` field that calls out the personality being tested.
**Cost:** scenario file duplication — each variant is a copy of the same rubric content with a different `name:` field. Cheap. The duplication is mechanical (you can clone the source scenario file 4-6 times with a one-line `name:` change each).
55
+
56
+
**Pays for itself the first time** you debug a failed run and need to identify which test variant flickered. Particularly important for LLM-as-judge sim suites where the per-evaluator results are subtle and the scenario name is your primary navigation handle.
57
+
58
+
**Anti-pattern:** putting one shared scenario behind N personality variants in the same suite. The dashboard sidebar shows N rows with identical scenario names, only distinguishable by clicking into each item to see the personality. Sim iteration time inflates because every failure investigation starts with "wait, which one was this?"
59
+
60
+
Cross-reference: this convention surfaced as friction during the Mudflap iForm Voicemail Triage sim iteration (PRISM-481). Original suites shipped with one shared scenario per group (4 live-pickup tests sharing one scenario, 6 voicemail-edge-cases sharing one scenario); split into per-personality scenarios mid-iteration. Worth shipping new suites in the per-personality form from day one.
0 commit comments