Replace single scoring policy with per-task scoring policies

ExerciseConfig now carries a per_task_scoring_policies JSON column keyed
by submission-test task name instead of a single scoring_policy dict.
Task names are AST-discovered from each exercise's submission_tests
file, so the config editor shows exactly the tasks registered by the
test script. Submissions are scored with score_submission(), which
applies each task's policy independently and returns (total, per-task
breakdown); the scoreboard API exposes the breakdown to the SPA.

The public scoreboard drops the user-selectable ranking strategy: the
SCOREBOARD_RANKING_MODE setting and RANKING_STRATEGIES registry are
removed, and the SPA is hardwired to the best-sum strategy. The chart
samples the timeline at real event timestamps only, so every plotted
point corresponds to an actual submission or window edge.

Author: Nils Bars

.claude/CLAUDE.md

@@ -23,9 +23,13 @@ The test infrastructure (`tests/helpers/ref_instance.py`) automatically sets thi
 # Build all Docker images
 ./ctrl.sh build
 
-# Start services (--debug attaches to terminal with logs)
-./ctrl.sh up --debug
-./ctrl.sh up
+# Start services
+# For development, always use --debug and --hot-reloading:
+#   --debug         enables Flask debug mode and verbose logging
+#   --hot-reloading enables Flask auto-reload and runs the spa-frontend
+#                   under `vite dev` (Vite HMR) instead of a static build
+./ctrl.sh up --debug --hot-reloading
+./ctrl.sh up                            # production-style start, no HMR
 
 # Stop services
 ./ctrl.sh stop              # Keep containers

README.md

@@ -184,4 +184,4 @@ The following features are disabled by default and can be enabled from the admin
 Allows students to be organized into named groups with a configurable maximum size. Students pick a group during registration, and admins can manage the available groups and reassign students afterwards. Enable via the `GROUPS_ENABLED` setting and configure the per-group capacity via `GROUP_SIZE`.
 
 #### Scoreboard
-A public leaderboard at `/v2/scoreboard` that ranks students based on their exercise submissions. Exercises can be grouped into assignments and the ranking strategy is selected via `SCOREBOARD_RANKING_MODE`. Enable via `SCOREBOARD_ENABLED`; optionally set `LANDING_PAGE` to `scoreboard` to use it as the default landing page.
+A public leaderboard at `/v2/scoreboard` that ranks students based on their exercise submissions using a Formula 1 style, time-weighted strategy. Exercises can be grouped into assignments. Enable via `SCOREBOARD_ENABLED`; optionally set `LANDING_PAGE` to `scoreboard` to use it as the default landing page.

docs/SCOREBOARD.md

@@ -3,10 +3,11 @@
 A public leaderboard at `/v2/scoreboard` that ranks students/teams based
 on submission scores. Exercises are grouped into **assignments**
 (time-boxed rounds, one per `ExerciseConfig.category`). Each exercise
-has a **scoring policy** that transforms raw submission scores into
-scoreboard points. The Vue SPA fetches metadata + submissions via two
-JSON endpoints and renders rankings, badges, charts, and per-challenge
-plots client-side.
+has **per-task scoring policies** that transform the raw score of each
+submission-test task into scoreboard points; the submission's total is
+the sum of the transformed per-task scores. The Vue SPA fetches metadata
++ submissions via two JSON endpoints and renders rankings, badges,
+charts, and per-challenge plots client-side.
 
 ## Data Model
 
@@ -22,7 +23,7 @@ class ExerciseConfig(db.Model):
     id: Mapped[int]                                     # PK
     short_name: Mapped[str]                             # unique
     category: Mapped[Optional[str]]                     # assignment label
-    scoring_policy: Mapped[Optional[dict]]              # JSON, see below
+    per_task_scoring_policies: Mapped[Optional[dict]]   # JSON: {task_name: policy}
     submission_deadline_start: Mapped[Optional[datetime]]
     submission_deadline_end: Mapped[Optional[datetime]]
     submission_test_enabled: Mapped[bool]
@@ -43,12 +44,21 @@ retroactively without reprocessing stored data.
 
 ## Scoring Policies
 
-The `scoring_policy` column on `ExerciseConfig` is a JSON object the
-admin edits from the exercise config page. `ref/core/scoring.py` exposes
-`apply_scoring(raw, policy)` which every API call routes raw scores
-through.
+`ExerciseConfig.per_task_scoring_policies` is a JSON object keyed by
+submission-test task name, where each value is a policy dict. The admin
+edits it from the exercise config page; task names are auto-discovered
+from the exercise's `submission_tests` file via AST parsing
+(`ref/core/task_discovery.py::extract_task_names_from_submission_tests`),
+so the editor always shows exactly the tasks the test script registers.
 
-Supported modes:
+`ref/core/scoring.py::score_submission(results, per_task_policies)`
+applies each task's policy (or pass-through if the task has no entry)
+to that task's raw score and returns `(total, breakdown)` where
+`breakdown[task_name]` is the transformed score (or `None` for tasks
+whose raw score was `None`). `total` sums the transformed scores;
+`None`-scored tasks contribute 0.
+
+Supported policy modes (same shape as `apply_scoring(raw, policy)`):
 
 ```
 # Linear mapping: raw [min_raw..max_raw] → [0..max_points]
@@ -71,29 +81,19 @@ tiers:
 ```
 
 Any policy may also carry an optional `baseline` field. It has no effect
-on the transformed score; the SPA renders it as a horizontal reference
-line on per-challenge plots (typically the score of a naive/trivial
-solution).
+on the transformed score; the SPA renders the **sum of per-task
+baselines** as a horizontal reference line on per-challenge plots
+(typically the score of a naive/trivial solution).
 
 `validate_scoring_policy(policy)` in the same module returns a list of
-human-readable error strings — the exercise-config edit view uses this
-to surface admin mistakes before persisting.
-
-## Ranking Strategies
+human-readable error strings for a single policy dict — the exercise-
+config edit view validates each per-task entry with it before persisting.
 
-Ranking strategies are registered in `RANKING_STRATEGIES` in
-`ref/core/scoring.py`. The active strategy is chosen by the
-`SCOREBOARD_RANKING_MODE` system setting and surfaced to the SPA via the
-config endpoint. Each strategy has a matching TypeScript module under
-`spa-frontend/src/ranking/` that computes the ranking client-side.
+## Ranking Strategy
 
-| Id | Label | Source |
-|----|-------|--------|
-| `f1_time_weighted` | Formula 1 (time-weighted) | `spa-frontend/src/ranking/f1_time_weighted.ts` |
-| `best_sum` | Sum of best per challenge | `spa-frontend/src/ranking/best_sum.ts` |
-
-Adding a strategy is one dict entry on the Python side plus one `.ts`
-file on the frontend.
+Ranking is computed client-side by
+`spa-frontend/src/ranking/f1_time_weighted.ts` — a Formula 1 style,
+time-weighted strategy. It is the only strategy the scoreboard supports.
 
 ## API Endpoints
 
@@ -103,46 +103,59 @@ feature never leaks its existence). No authentication required.
 
 ### `GET /api/scoreboard/config`
 
-Assignment/challenge metadata plus the active ranking strategy.
+Assignment/challenge metadata.
 
 ```json
 {
   "course_name": "OS-Security",
-  "ranking_mode": "f1_time_weighted",
   "assignments": {
     "Assignment 1": {
       "exercise_short_name": {
         "start": "DD/MM/YYYY HH:MM:SS",
         "end":   "DD/MM/YYYY HH:MM:SS",
-        "scoring": { "mode": "threshold", "threshold": 0.5, "points": 100, "baseline": 0.013 },
-        "max_points": 100
+        "per_task_scoring_policies": {
+          "coverage": { "mode": "linear", "max_points": 100, "baseline": 0.013 },
+          "crashes":  { "mode": "threshold", "threshold": 1, "points": 50 }
+        },
+        "max_points": 150
       }
     }
   }
 }
 ```
 
-Only exercises whose default version has finished building and whose
-`ExerciseConfig` has both deadline endpoints + a non-null `category` are
-included. Empty assignment buckets are pruned.
+`max_points` is the best-effort sum of each per-task policy's upper
+bound (used by the frontend for axis scaling); it is `null` if no task
+has a computable maximum. Only exercises whose default version has
+finished building and whose `ExerciseConfig` has both deadline endpoints
++ a non-null `category` are included. Empty assignment buckets are
+pruned.
 
 ### `GET /api/scoreboard/submissions`
 
-Submission scores grouped by exercise and team, pre-transformed by
-`apply_scoring()`:
+Submission scores grouped by exercise and team, pre-transformed via
+`score_submission()` with a per-task breakdown:
 
 ```json
 {
   "exercise_short_name": {
-    "Team A": [["DD/MM/YYYY HH:MM:SS", 87.5], ...]
+    "Team A": [
+      {
+        "ts": "DD/MM/YYYY HH:MM:SS",
+        "score": 87.5,
+        "tasks": { "coverage": 50.0, "crashes": 37.5, "env_check": null }
+      }
+    ]
   }
 }
 ```
 
-Submissions with zero or multiple test results are skipped and logged;
-the endpoint expects exactly one top-level test result per submission.
-The team label comes from `team_identity(user)`, which returns the
-user's group name when groups are enabled, otherwise their full name.
+`tasks` values of `null` mean the underlying `SubmissionTestResult.score`
+was `None` (bool-returning test, no grading) — consumers render these
+as "untested" rather than 0. Such tasks contribute 0 to the outer
+`score`. Submissions with no test results at all are skipped. The team
+label comes from `team_identity(user)`, which returns the user's group
+name when groups are enabled, otherwise their full name.
 
 ## Frontend
 
@@ -172,8 +185,7 @@ dedicated backend. Badge assets are static SVG files at
 | Setting | Type | Purpose |
 |---------|------|---------|
 | `SCOREBOARD_ENABLED` | bool | Master toggle for the page + JSON endpoints |
-| `SCOREBOARD_RANKING_MODE` | str | Selected ranking strategy id |
 | `LANDING_PAGE` | str | `"registration"` or `"scoreboard"` — where `/` redirects |
 
-All three are exposed in the admin system-settings form
+Both are exposed in the admin system-settings form
 (`webapp/ref/view/system_settings.py`).

spa-frontend/src/api/scoreboard.ts

@@ -1,23 +1,35 @@
 import { apiGet } from './client';
 
+// Policy shape mirrors ref/core/scoring.py::apply_scoring inputs.
+export type ScoringPolicy = Record<string, unknown> & { baseline?: number };
+
 // Mirrors /api/scoreboard/config response shape.
 export interface ChallengeCfg {
   start: string;
   end: string;
-  scoring: Record<string, unknown> & { baseline?: number };
+  per_task_scoring_policies: Record<string, ScoringPolicy>;
   max_points: number | null;
 }
 
 export type Assignments = Record<string, Record<string, ChallengeCfg>>;
 
 export interface ScoreboardConfig {
   course_name: string;
-  ranking_mode: string;
   assignments: Assignments;
 }
 
-// Submissions: challenge -> team -> [[tsStr, score], ...]
-export type TeamSubmissions = Record<string, Array<[string, number]>>;
+// One submission entry as returned by /api/scoreboard/submissions.
+// `tasks` maps task_name -> transformed score; `null` means the task's
+// raw score was None (bool-returning test) and should be rendered as
+// "untested" rather than zero.
+export interface SubmissionEntry {
+  ts: string;
+  score: number;
+  tasks: Record<string, number | null>;
+}
+
+// Submissions: challenge -> team -> SubmissionEntry[]
+export type TeamSubmissions = Record<string, SubmissionEntry[]>;
 export type SubmissionsByChallenge = Record<string, TeamSubmissions>;
 
 export function getScoreboardConfig(): Promise<ScoreboardConfig> {

spa-frontend/src/components/scoreboard/ChallengePlot.vue

@@ -18,27 +18,49 @@ let chart: Chart | null = null;
 let xMinCache = 0;
 
 function findBaseline(): number | null {
+  // The plot's baseline line is drawn at the sum of per-task baselines
+  // (each task's policy may optionally carry one). Returns null if no
+  // task has a baseline configured.
   for (const challenges of Object.values(props.assignments || {})) {
     const cfg = challenges[props.challengeName];
-    if (cfg && cfg.scoring && typeof cfg.scoring.baseline === 'number') {
-      return cfg.scoring.baseline;
+    if (!cfg || !cfg.per_task_scoring_policies) continue;
+    let total = 0;
+    let any = false;
+    for (const policy of Object.values(cfg.per_task_scoring_policies)) {
+      if (policy && typeof policy.baseline === 'number') {
+        total += policy.baseline;
+        any = true;
+      }
     }
+    if (any) return total;
   }
   return null;
 }
 
+type PlotPoint = {
+  x: number;
+  y: number;
+  tasks: Record<string, number | null>;
+};
+
 // eslint-disable-next-line @typescript-eslint/no-explicit-any
 function buildDatasets(): any[] {
   const teams = (props.submissions && props.submissions[props.challengeName]) || {};
   return Object.entries(teams).map(([team, points]) => {
     const parsed = points
-      .map(([tsStr, score]) => {
-        const d = parseApiDate(tsStr);
-        return d ? { x: d.getTime(), y: Number(score) } : null;
+      .map((entry) => {
+        const d = parseApiDate(entry.ts);
+        return d
+          ? {
+              x: d.getTime(),
+              y: Number(entry.score),
+              tasks: entry.tasks ?? {},
+            }
+          : null;
       })
-      .filter((p): p is { x: number; y: number } => p !== null)
+      .filter((p): p is PlotPoint => p !== null)
       .sort((a, b) => a.x - b.x);
-    const improvements: { x: number; y: number }[] = [];
+    const improvements: PlotPoint[] = [];
     let best = -Infinity;
     for (const p of parsed) {
       if (p.y > best) {
@@ -105,6 +127,27 @@ function render() {
         annotation: { annotations },
         legend: { labels: { usePointStyle: true } },
         zoom: makeZoomPanOptions(() => xMinCache),
+        tooltip: {
+          callbacks: {
+            // eslint-disable-next-line @typescript-eslint/no-explicit-any
+            label: (ctx: any) =>
+              `${ctx.dataset.label ?? ''}: ${Number(ctx.parsed.y).toFixed(2)}`,
+            // eslint-disable-next-line @typescript-eslint/no-explicit-any
+            afterBody: (items: any[]) => {
+              if (!items.length) return [];
+              const raw = items[0].raw as PlotPoint | undefined;
+              const tasks = raw?.tasks;
+              if (!tasks || Object.keys(tasks).length < 2) return [];
+              const lines = ['', 'Tasks:'];
+              for (const [name, score] of Object.entries(tasks)) {
+                const rendered =
+                  score === null ? 'untested' : Number(score).toFixed(2);
+                lines.push(`  ${name}: ${rendered}`);
+              }
+              return lines;
+            },
+          },
+        },
       },
     },
   });

spa-frontend/src/components/scoreboard/PointsOverTimeChart.vue

@@ -47,7 +47,17 @@ function buildAnnotations(): Record<string, any> {
       borderWidth: 1,
       scaleID: 'x',
       value: t.getTime(),
-      label: { content: `Assignment ${i + 1}`, display: true },
+      label: {
+        content: `Assignment ${i + 1}`,
+        display: true,
+        rotation: -90,
+        position: 'center',
+        xAdjust: 12,
+        yAdjust: -10,
+        backgroundColor: 'rgba(0, 0, 0, 0)',
+        color: 'gray',
+        padding: 0,
+      },
     };
   });
   return annotations;

spa-frontend/src/components/scoreboard/RankingTable.vue

@@ -2,10 +2,14 @@
 import type { Ranking } from '../../ranking/types';
 import type { Badges } from '../../ranking/util';
 
-defineProps<{
-  ranking: Ranking;
-  badges: Badges;
-}>();
+const props = withDefaults(
+  defineProps<{
+    ranking: Ranking;
+    badges?: Badges;
+    hideBadges?: boolean;
+  }>(),
+  { badges: () => ({}), hideBadges: false },
+);
 
 function badgeSrc(name: string): string {
   return `/static/badges/${name}.svg`;
@@ -16,6 +20,8 @@ function onBadgeError(e: Event) {
   img.onerror = null;
   img.src = '/static/badges/default.svg';
 }
+
+const colSpan = props.hideBadges ? 3 : 4;
 </script>
 
 <template>
@@ -25,18 +31,18 @@ function onBadgeError(e: Event) {
         <tr>
           <th class="term-col-rank">#</th>
           <th>Team</th>
-          <th>Badges</th>
+          <th v-if="!hideBadges">Badges</th>
           <th class="term-col-points">Points</th>
         </tr>
       </thead>
       <tbody>
         <tr v-if="!ranking || ranking.length === 0" class="term-empty">
-          <td colspan="4">// awaiting submissions</td>
+          <td :colspan="colSpan">// awaiting submissions</td>
         </tr>
         <tr v-for="([team, score], i) in ranking" :key="team">
           <td class="term-col-rank"><span class="term-rank">{{ i + 1 }}</span></td>
           <td class="term-team">{{ team }}</td>
-          <td>
+          <td v-if="!hideBadges">
             <span class="term-badges">
               <img
                 v-for="b in badges[team] || []"