You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: .github/workflows/workshop-student-simulator.md
+6-15Lines changed: 6 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -392,33 +392,24 @@ Use `monteCarlo[*].successRate` as the **statistical success probability** for e
392
392
393
393
Use `successRateCi95`, `aggregate.overallSuccessRateCi95`, and `aggregate.dropoutRateCi95ByStep` to show Monte Carlo uncertainty. These intervals do not cover uncertainty in the assumed population weights or probability model; repeat that limitation in the report. Use `aggregate.attemptsByStep` as each step's at-risk denominator. `aggregate.dropoutRateByStep` is the conditional probability of failing among runs that reached the step, not a percentage of all starting runs.
394
394
395
-
Then, for each student, use the environment assumptions modelled by the simulator to explain **why** their success rate is what it is. Read the student's Monte Carlo entry (`failuresByStep`, `mostCommonFailureStep`), inspect the matching `stepContentById[mostCommonFailureStep]`, and cross-reference both with the student profile to produce per-student pain points:
395
+
### Collect pain points per student
396
396
397
-
- Which step failed most often across the ${{ env.MONTE_CARLO_RUNS }} runs
398
-
- The likely environment or profile reason (OS, tool, auth, level, personality)
399
-
- The likely content reason (for example: terminal-heavy instructions, browser-friendly fallback, auth-heavy setup, or high conceptual density)
397
+
For each student whose `successRate` < 1.0, note:
398
+
- Which step failed most often across the ${{ env.MONTE_CARLO_RUNS }} Monte Carlo runs (`mostCommonFailureStep`)
399
+
- The failure count per step from `failuresByStep`
400
+
- Likely reason (based on profile **and** content): reason from the student's `level`, `background`, `personality`, `tool`, `mobile`, and `ui_preferred` in relation to `stepContentById[step]` and the step's actual content and demands. Do **not** match against a fixed template. Key edge cases to flag explicitly: `ui_preferred: true` or `mobile: true` students hitting terminal-only steps (no UI alternative exists); Codespaces learners choosing the optional CLI trigger path and hitting `actions:write` friction; enterprise/proxy environments adding friction to setup steps.
400
401
- Treat browser-driven workflow execution steps differently from local CLI steps: triggering a workflow from the **Actions** tab should not require local Copilot credentials. Only flag secret-related problems at that stage when `aggregate.failureCategoriesByStep` reports that exact runtime failure after the learner completed the preceding model-access activity.
401
402
- Do not infer a failure reason from lexical signals such as `authDemand`. The baseline first workflow uses GitHub Copilot; do not introduce optional engines or credentials from later side quests into its failure analysis. Use `failureCategoriesByStep` as the source of truth for the top reason.
402
403
- For `tool: CCA` learners, treat the Agents tab as a prompt surface. If the step tells the learner to run shell commands there, or fails to call out `/agentic-workflows` for workflow-authoring work, classify that as a content mismatch rather than a generic terminal-skill gap.
403
404
404
-
#### Qualitative depth for top-failure students
405
-
406
-
For students whose `successRate` < 0.50 (the most at-risk half), apply additional qualitative reasoning from the student profile to enrich the pain-point description:
407
-
405
+
For students whose `successRate` < 0.50, also apply qualitative reasoning from the student profile to enrich the pain-point description:
408
406
-**`level`** vs. assumed knowledge
409
407
-**`background`** vs. step domain
410
408
-**`tool`**, **`mobile`**, and **`ui_preferred`** vs. step tooling (if a step requires running `gh aw` in a terminal and the student is `ui_preferred`, `mobile: true`, or uses `CCA`, note that no UI alternative exists)
- Which step failed most often across the ${{ env.MONTE_CARLO_RUNS }} Monte Carlo runs (`mostCommonFailureStep`)
419
-
- The failure count per step from `failuresByStep`
420
-
- Likely reason (based on profile **and** content): reason from the student's `level`, `background`, `personality`, `tool`, `mobile`, and `ui_preferred` in relation to `stepContentById[step]` and the step's actual content and demands. Do **not** match against a fixed template. Key edge cases to flag explicitly: `ui_preferred: true` or `mobile: true` students hitting terminal-only steps (no UI alternative exists); Codespaces learners choosing the optional CLI trigger path and hitting `actions:write` friction; enterprise/proxy environments adding friction to setup steps.
421
-
422
413
### Aggregate and analyse results
423
414
424
415
Use the pre-computed values from `monte-carlo-replay.json` as the primary data source:
0 commit comments