Skip to content

Commit e2adc94

Browse files
Copilotpelikhan
andauthored
refactor(simulator): consolidate duplicate pain-point instructions into single section (#1755)
* Initial plan * refactor: merge duplicate pain-point sections in workshop-student-simulator Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * fix: restore stable cron schedule via --schedule-seed recompile Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
1 parent 7b898c0 commit e2adc94

2 files changed

Lines changed: 7 additions & 16 deletions

File tree

.github/workflows/workshop-student-simulator.lock.yml

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/workshop-student-simulator.md

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -392,33 +392,24 @@ Use `monteCarlo[*].successRate` as the **statistical success probability** for e
392392

393393
Use `successRateCi95`, `aggregate.overallSuccessRateCi95`, and `aggregate.dropoutRateCi95ByStep` to show Monte Carlo uncertainty. These intervals do not cover uncertainty in the assumed population weights or probability model; repeat that limitation in the report. Use `aggregate.attemptsByStep` as each step's at-risk denominator. `aggregate.dropoutRateByStep` is the conditional probability of failing among runs that reached the step, not a percentage of all starting runs.
394394

395-
Then, for each student, use the environment assumptions modelled by the simulator to explain **why** their success rate is what it is. Read the student's Monte Carlo entry (`failuresByStep`, `mostCommonFailureStep`), inspect the matching `stepContentById[mostCommonFailureStep]`, and cross-reference both with the student profile to produce per-student pain points:
395+
### Collect pain points per student
396396

397-
- Which step failed most often across the ${{ env.MONTE_CARLO_RUNS }} runs
398-
- The likely environment or profile reason (OS, tool, auth, level, personality)
399-
- The likely content reason (for example: terminal-heavy instructions, browser-friendly fallback, auth-heavy setup, or high conceptual density)
397+
For each student whose `successRate` < 1.0, note:
398+
- Which step failed most often across the ${{ env.MONTE_CARLO_RUNS }} Monte Carlo runs (`mostCommonFailureStep`)
399+
- The failure count per step from `failuresByStep`
400+
- Likely reason (based on profile **and** content): reason from the student's `level`, `background`, `personality`, `tool`, `mobile`, and `ui_preferred` in relation to `stepContentById[step]` and the step's actual content and demands. Do **not** match against a fixed template. Key edge cases to flag explicitly: `ui_preferred: true` or `mobile: true` students hitting terminal-only steps (no UI alternative exists); Codespaces learners choosing the optional CLI trigger path and hitting `actions:write` friction; enterprise/proxy environments adding friction to setup steps.
400401
- Treat browser-driven workflow execution steps differently from local CLI steps: triggering a workflow from the **Actions** tab should not require local Copilot credentials. Only flag secret-related problems at that stage when `aggregate.failureCategoriesByStep` reports that exact runtime failure after the learner completed the preceding model-access activity.
401402
- Do not infer a failure reason from lexical signals such as `authDemand`. The baseline first workflow uses GitHub Copilot; do not introduce optional engines or credentials from later side quests into its failure analysis. Use `failureCategoriesByStep` as the source of truth for the top reason.
402403
- For `tool: CCA` learners, treat the Agents tab as a prompt surface. If the step tells the learner to run shell commands there, or fails to call out `/agentic-workflows` for workflow-authoring work, classify that as a content mismatch rather than a generic terminal-skill gap.
403404

404-
#### Qualitative depth for top-failure students
405-
406-
For students whose `successRate` < 0.50 (the most at-risk half), apply additional qualitative reasoning from the student profile to enrich the pain-point description:
407-
405+
For students whose `successRate` < 0.50, also apply qualitative reasoning from the student profile to enrich the pain-point description:
408406
- **`level`** vs. assumed knowledge
409407
- **`background`** vs. step domain
410408
- **`tool`**, **`mobile`**, and **`ui_preferred`** vs. step tooling (if a step requires running `gh aw` in a terminal and the student is `ui_preferred`, `mobile: true`, or uses `CCA`, note that no UI alternative exists)
411409
- **`personality`**: `methodical` reads carefully; `confused` needs more guidance; `impatient` skips steps
412410
- **`goal`**: `team-evaluation` abandons sooner; `teaching-others` is thorough
413411
- **Prior runs** (`runs`, `successes`): higher prior completions correlate with better outcomes
414412

415-
### Collect pain points per student
416-
417-
For each student whose `successRate` < 1.0, note:
418-
- Which step failed most often across the ${{ env.MONTE_CARLO_RUNS }} Monte Carlo runs (`mostCommonFailureStep`)
419-
- The failure count per step from `failuresByStep`
420-
- Likely reason (based on profile **and** content): reason from the student's `level`, `background`, `personality`, `tool`, `mobile`, and `ui_preferred` in relation to `stepContentById[step]` and the step's actual content and demands. Do **not** match against a fixed template. Key edge cases to flag explicitly: `ui_preferred: true` or `mobile: true` students hitting terminal-only steps (no UI alternative exists); Codespaces learners choosing the optional CLI trigger path and hitting `actions:write` friction; enterprise/proxy environments adding friction to setup steps.
421-
422413
### Aggregate and analyse results
423414

424415
Use the pre-computed values from `monte-carlo-replay.json` as the primary data source:

0 commit comments

Comments
 (0)