|
| 1 | +# Scoreboard Integration |
| 2 | + |
| 3 | +This document describes how to integrate a scoreboard into REF, based on the prototype in the `raid/raid` branch and adapted to the current `dev` codebase. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The scoreboard is a public-facing page that shows team/student rankings based on submission scores. Exercises are grouped into **waves** (time-boxed rounds). Each exercise defines a **scoring policy** that maps raw submission scores to scoreboard points. The frontend fetches data via two JSON APIs and renders rankings, badges, charts, and per-challenge plots client-side. |
| 8 | + |
| 9 | +## What exists in `raid/raid` |
| 10 | + |
| 11 | +The prototype adds: |
| 12 | + |
| 13 | +- **Two API endpoints** (`/api/waves`, `/api/submissions`) that return exercise metadata and submission scores as JSON. |
| 14 | +- **Three new Exercise model fields**: `baseline_score`, `badge_score`, `badge_points` — parsed from the exercise YAML config. |
| 15 | +- **A scoreboard page** (`/student/scoreboard`) with a Jinja template and ~2300 lines of client-side JS (`scoreboard.js`, `utils.js`, `plots.js`) using Chart.js. |
| 16 | +- **Badges**: per-challenge achievement icons shown in the ranking table when a team's score exceeds `badge_score`. Each challenge can have custom SVG/PNG assets with a default fallback. |
| 17 | +- **System settings**: `LANDING_PAGE` (choose which page students see first), `DEMO_MODE_ENABLED` (serve dummy JSON data). |
| 18 | +- **A demo/dummy data system** (`dummies/waves.json`, `dummies/submissions.json`) for development without real submissions. |
| 19 | + |
| 20 | +The prototype is tightly coupled to a fixed 3-waves x 3-challenges layout and hardcodes wave/challenge indices in the template. It also bundles all scoring logic (ranking, badges, rates) in the frontend JS. |
| 21 | + |
| 22 | +## What already exists in `dev` |
| 23 | + |
| 24 | +- `SubmissionTestResult.score` (float, nullable) — already in the model. This is the raw per-submission score. |
| 25 | +- Exercise `category` field — used as the wave/group name. |
| 26 | +- `Submission.all()`, exercise deadlines, the full submission lifecycle. |
| 27 | + |
| 28 | +## ExerciseConfig: Separating Administrative from Build-Time Config |
| 29 | + |
| 30 | +### Problem |
| 31 | + |
| 32 | +Currently, all exercise configuration lives on the `Exercise` model — one row per version. Administrative fields like deadlines, category, and max grading points are duplicated across versions and synchronized at import time (importing a new version propagates deadlines to all predecessors). This is fragile: editing via a web UI would require updating every version row. |
| 33 | + |
| 34 | +### Solution: `ExerciseConfig` Model |
| 35 | + |
| 36 | +Introduce a new model that holds **administrative configuration** shared across all versions of an exercise: |
| 37 | + |
| 38 | +```python |
| 39 | +class ExerciseConfig(db.Model): |
| 40 | + __tablename__ = 'exercise_config' |
| 41 | + |
| 42 | + id: Mapped[int] # PK (integer) |
| 43 | + short_name: Mapped[str] # unique constraint |
| 44 | + category: Mapped[Optional[str]] # wave/group name |
| 45 | + scoring_policy: Mapped[Optional[dict]] # JSON, see Scoring Architecture |
| 46 | + submission_deadline_start: Mapped[Optional[datetime]] |
| 47 | + submission_deadline_end: Mapped[Optional[datetime]] |
| 48 | + submission_test_enabled: Mapped[bool] |
| 49 | + max_grading_points: Mapped[Optional[int]] |
| 50 | +``` |
| 51 | + |
| 52 | +The `Exercise` model gets a FK to `ExerciseConfig`: |
| 53 | + |
| 54 | +```python |
| 55 | +class Exercise(db.Model): |
| 56 | + # ... existing build-time fields ... |
| 57 | + config_id: Mapped[int] = mapped_column(ForeignKey('exercise_config.id')) |
| 58 | + config: Mapped[ExerciseConfig] = relationship(...) |
| 59 | +``` |
| 60 | + |
| 61 | +All versions of an exercise with the same `short_name` point to the **same** `ExerciseConfig` row. |
| 62 | + |
| 63 | +### What stays on `Exercise` (per-version, build-time) |
| 64 | + |
| 65 | +- `entry_service` — files, build commands, cmd, flags, resource limits, ASLR, readonly, networking |
| 66 | +- `services` — peripheral service configs |
| 67 | +- `build_job_status` / `build_job_result` |
| 68 | +- `template_path` / `persistence_path` / `template_import_path` |
| 69 | +- `is_default` |
| 70 | +- `version` |
| 71 | + |
| 72 | +### What moves to `ExerciseConfig` (global, web-editable) |
| 73 | + |
| 74 | +- `category` |
| 75 | +- `submission_deadline_start` / `submission_deadline_end` |
| 76 | +- `submission_test_enabled` |
| 77 | +- `max_grading_points` |
| 78 | +- `scoring_policy` (new) |
| 79 | + |
| 80 | +### How it integrates with versioning |
| 81 | + |
| 82 | +- **First import of an exercise:** Creates an `ExerciseConfig` row. Initial values come from the YAML config. |
| 83 | +- **Reimport (new version):** Reuses the existing `ExerciseConfig` (looked up by `short_name`). The new `Exercise` row points to the same config. YAML values for administrative fields are **ignored** — the web-edited config takes precedence. |
| 84 | +- **Web UI edit:** Updates the single `ExerciseConfig` row — immediately effective for all versions. |
| 85 | +- **No more sync logic:** Deadlines no longer need to be propagated across version rows on import. |
| 86 | + |
| 87 | +### Migration path for other config |
| 88 | + |
| 89 | +As more settings move from YAML to web UI, the pattern is: |
| 90 | +1. **Build-time?** → stays on `Exercise` (per-version, immutable after build) |
| 91 | +2. **Administrative?** → moves to `ExerciseConfig` (global, web-editable) |
| 92 | + |
| 93 | +Future candidates for `ExerciseConfig`: display name, description, visibility/published flag, ordering/priority. |
| 94 | + |
| 95 | +### Web UI: Edit Button |
| 96 | + |
| 97 | +The exercise list page gets an **Edit** button per exercise (per `short_name`, not per version). It opens a form editing the `ExerciseConfig`: |
| 98 | + |
| 99 | +- Category / wave assignment |
| 100 | +- Deadlines (start, end) |
| 101 | +- Scoring policy (mode selector + mode-specific fields) |
| 102 | +- Max grading points |
| 103 | +- Submission test toggle |
| 104 | + |
| 105 | +This is separate from the import flow which handles build-time config. |
| 106 | + |
| 107 | +## Scoring Architecture |
| 108 | + |
| 109 | +### Raw Scores vs. Scoreboard Points |
| 110 | + |
| 111 | +Submissions produce a **raw score** (float, stored in `SubmissionTestResult.score`). This raw score needs to be translated into **scoreboard points** via the exercise's **scoring policy** (stored on `ExerciseConfig`). |
| 112 | + |
| 113 | +### Scoring Policy |
| 114 | + |
| 115 | +The scoring policy is configured via the web UI and stored as a JSON column on `ExerciseConfig`. Supported modes: |
| 116 | + |
| 117 | +``` |
| 118 | +# Linear mapping: raw [0..1] → [0..max_points] |
| 119 | +mode: linear |
| 120 | +max_points: 100 |
| 121 | +
|
| 122 | +# Threshold: all-or-nothing |
| 123 | +mode: threshold |
| 124 | +threshold: 0.5 |
| 125 | +points: 100 |
| 126 | +
|
| 127 | +# Tiered: stepped milestones |
| 128 | +mode: tiered |
| 129 | +tiers: |
| 130 | + - above: 0.3, points: 25 |
| 131 | + - above: 0.6, points: 50 |
| 132 | + - above: 0.9, points: 100 |
| 133 | +``` |
| 134 | + |
| 135 | +An optional `baseline` field can be included in any mode to show a reference line on charts (e.g., the score of a naive/trivial solution). |
| 136 | + |
| 137 | +### Where Scoring is Evaluated |
| 138 | + |
| 139 | +**Server-side, in core logic.** The server applies the scoring policy when serving the submissions API. Reasons: |
| 140 | + |
| 141 | +- **Authoritative ranking** — no client-side inconsistencies. |
| 142 | +- **Retroactive changes** — changing a policy recomputes on the fly since raw scores are stored, not transformed ones. |
| 143 | +- **Single source of truth** — one evaluation function in `ref/core/`. |
| 144 | + |
| 145 | +### Badges |
| 146 | + |
| 147 | +Badges are a **visual consequence of scoring**, not a separate system. When a team earns points for a challenge (i.e., crosses a threshold or achieves a score), the frontend renders a badge icon. Badge assets are static files per exercise name (`/static/badges/<name>.svg`) with a default fallback. No extra backend logic is needed — the frontend derives badges from the scoring data. |
| 148 | + |
| 149 | +## Integration Plan |
| 150 | + |
| 151 | +### 1. `ExerciseConfig` Model and Migration |
| 152 | + |
| 153 | +Create the `ExerciseConfig` model. Migrate existing administrative fields (`category`, deadlines, `submission_test_enabled`, `max_grading_points`) from `Exercise` rows into `ExerciseConfig` rows. Add `scoring_policy` JSON column. Update `Exercise` with a FK `config_id` pointing to `ExerciseConfig`. |
| 154 | + |
| 155 | +The migration: |
| 156 | +1. Create `exercise_config` table. |
| 157 | +2. Populate it from distinct `short_name` values in `exercise`, taking field values from the head (newest) version. |
| 158 | +3. Add `config_id` FK column to `exercise` and backfill it. |
| 159 | +4. (Optional, later) Drop the migrated columns from `exercise`. |
| 160 | + |
| 161 | +### 2. Update ExerciseManager |
| 162 | + |
| 163 | +- On first import: create `ExerciseConfig` from YAML values. |
| 164 | +- On reimport: look up existing `ExerciseConfig` by `short_name`, skip administrative fields from YAML. |
| 165 | +- Remove the deadline sync logic from `check_global_constraints()`. |
| 166 | + |
| 167 | +### 3. Scoring API Endpoints |
| 168 | + |
| 169 | +**`GET /api/scoreboard/config`** — Returns exercise metadata grouped by `category` (wave), including the scoring policy: |
| 170 | + |
| 171 | +```json |
| 172 | +{ |
| 173 | + "Wave 1": { |
| 174 | + "exercise_name": { |
| 175 | + "start": "...", |
| 176 | + "end": "...", |
| 177 | + "scoring": { |
| 178 | + "mode": "threshold", |
| 179 | + "threshold": 0.5, |
| 180 | + "points": 100, |
| 181 | + "baseline": 0.013 |
| 182 | + } |
| 183 | + } |
| 184 | + } |
| 185 | +} |
| 186 | +``` |
| 187 | + |
| 188 | +**`GET /api/submissions`** — Returns transformed submission scores grouped by exercise and team/user: |
| 189 | + |
| 190 | +```json |
| 191 | +{ |
| 192 | + "exercise_name": { |
| 193 | + "Team A": [[timestamp, score], ...] |
| 194 | + } |
| 195 | +} |
| 196 | +``` |
| 197 | + |
| 198 | +Scores returned here are already transformed by the server using the exercise's scoring policy. |
| 199 | + |
| 200 | +Both endpoints are rate-limited and publicly accessible (no auth required). |
| 201 | + |
| 202 | +### 4. Exercise Edit UI |
| 203 | + |
| 204 | +Add an edit button to the exercise list page. The edit form modifies `ExerciseConfig` fields: |
| 205 | + |
| 206 | +- Category / wave |
| 207 | +- Deadlines |
| 208 | +- Scoring policy (mode dropdown + dynamic fields per mode) |
| 209 | +- Max grading points |
| 210 | +- Submission test toggle |
| 211 | + |
| 212 | +### 5. Scoreboard Frontend |
| 213 | + |
| 214 | +Add a scoreboard page at `/scoreboard`: |
| 215 | + |
| 216 | +- Fetches `/api/scoreboard/config` and `/api/submissions` periodically. |
| 217 | +- Renders a **ranking table** (sorted by total points) with **badge icons** for earned challenges. |
| 218 | +- Renders **per-challenge score charts** using Chart.js with baseline annotation lines. |
| 219 | +- Shows a **countdown timer** for the active wave's deadline. |
| 220 | +- Supports multiple waves via tab navigation. |
| 221 | +- Fully dynamic — number of waves and challenges driven by API data. |
| 222 | + |
| 223 | +The `raid/raid` JS can be reused but should be refactored to remove the hardcoded 3x3 layout. |
| 224 | + |
| 225 | +### 6. System Settings |
| 226 | + |
| 227 | +| Setting | Type | Purpose | |
| 228 | +|---------|------|---------| |
| 229 | +| `SCOREBOARD_ENABLED` | bool | Toggle scoreboard visibility | |
| 230 | +| `LANDING_PAGE` | str | Choose default student landing page (registration / scoreboard) | |
| 231 | + |
| 232 | +## Key Design Decisions (to be made) |
| 233 | + |
| 234 | +- **Public vs. authenticated scoreboard**: Should the scoreboard require login? The `raid/raid` version is public. |
| 235 | +- **Score aggregation**: Should ranking use sum of best scores per challenge, or sum of all earned points? The prototype sums badge points. |
| 236 | +- **Polling interval**: The prototype polls every 5 seconds. Consider server-side caching or a longer interval. |
| 237 | +- **Admin scoreboard controls**: Should admins be able to freeze/reset the scoreboard? |
| 238 | +- **Dropping old columns**: When to remove the migrated fields from `Exercise` (can be deferred to avoid a big-bang migration). |
| 239 | + |
| 240 | +## Files to Create/Modify |
| 241 | + |
| 242 | +| File | Action | |
| 243 | +|------|--------| |
| 244 | +| `webapp/ref/model/exercise_config.py` | New: `ExerciseConfig` model | |
| 245 | +| `webapp/ref/model/exercise.py` | Add `config_id` FK, remove migrated fields (later) | |
| 246 | +| `webapp/ref/core/exercise.py` | Update import logic to use `ExerciseConfig` | |
| 247 | +| `webapp/ref/core/scoring.py` | New: `apply_scoring()` helper | |
| 248 | +| `webapp/ref/view/api.py` | Add `/api/scoreboard/config` and `/api/submissions` endpoints | |
| 249 | +| `webapp/ref/view/exercise.py` | Add edit endpoint for `ExerciseConfig` | |
| 250 | +| `webapp/ref/templates/exercise_edit.html` | New: edit form template | |
| 251 | +| `webapp/ref/view/student.py` | Add scoreboard route | |
| 252 | +| `webapp/ref/templates/student_scoreboard.html` | New template | |
| 253 | +| `webapp/ref/static/js/scoreboard.js` | Adapt from `raid/raid` | |
| 254 | +| `webapp/ref/static/js/plots.js` | Adapt from `raid/raid` (Chart.js plots) | |
| 255 | +| `webapp/ref/static/js/utils.js` | Adapt from `raid/raid` (scoring logic) | |
| 256 | +| `webapp/ref/static/badges/` | Badge SVG assets (per exercise + default) | |
| 257 | +| `webapp/ref/model/settings.py` | Add `SCOREBOARD_ENABLED`, `LANDING_PAGE` | |
| 258 | +| `webapp/ref/view/system_settings.py` | Expose new settings in admin UI | |
| 259 | +| `migrations/versions/xxx_exercise_config.py` | DB migration | |
0 commit comments