Skip to content

Commit 39c93d6

Browse files
ArthurCRodriguesArthurCRodrigues
authored
Merge pull request #297 from webtech-network/285-external-mode-github-action-inputs-mode-plumbing-and-compatibility
Add external mode for GitHub Action
2 parents 157d763 + 9e1e11d commit 39c93d6

20 files changed

Lines changed: 2581 additions & 119 deletions

‎action.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,26 @@ inputs:
2828
description: 'Whether to include/generate feedback. Use "true" or "false". Defaults to false.'
2929
required: false
3030
default: "false"
31+
execution-mode:
32+
description: 'Execution mode for the autograder. Use "repo" (default) for repository-config mode or "external" to load config from the Autograder Cloud.'
33+
required: false
34+
default: "repo"
35+
grading-config-id:
36+
description: "The grading configuration ID from the Autograder Cloud. Required when execution-mode is \"external\"."
37+
required: false
38+
autograder-cloud-url:
39+
description: "The base URL of the Autograder Cloud instance. Required when execution-mode is \"external\"."
40+
required: false
41+
autograder-cloud-token:
42+
description: "Authentication token for the Autograder Cloud API. Required when execution-mode is \"external\"."
43+
required: false
44+
submission-language:
45+
description: "Programming language of the student submission (e.g. 'python', 'java'). Validated against the grading config's supported languages. Defaults to the first language in the config when omitted. Only used in external mode."
46+
required: false
47+
locale:
48+
description: "Locale for feedback messages (e.g. 'en', 'pt-br'). Defaults to 'en'."
49+
required: false
50+
default: "en"
3151

3252
# Defines the output variable of the action.
3353
outputs:
@@ -47,3 +67,9 @@ runs:
4767
OPENAI_KEY: ${{ inputs.openai-key }}
4868
APP_TOKEN: ${{ inputs.app-token }}
4969
INCLUDE_FEEDBACK: ${{ inputs.include-feedback }}
70+
EXECUTION_MODE: ${{ inputs.execution-mode }}
71+
GRADING_CONFIG_ID: ${{ inputs.grading-config-id }}
72+
AUTOGRADER_CLOUD_URL: ${{ inputs.autograder-cloud-url }}
73+
AUTOGRADER_CLOUD_TOKEN: ${{ inputs.autograder-cloud-token }}
74+
SUBMISSION_LANGUAGE: ${{ inputs.submission-language }}
75+
LOCALE: ${{ inputs.locale }}

‎autograder/models/abstract/exporter.py‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
"""Abstract interface for grade exporters."""
22

3+
from __future__ import annotations
4+
35
from abc import ABC, abstractmethod
4-
from typing import Optional
6+
from typing import Optional, TYPE_CHECKING
7+
8+
if TYPE_CHECKING:
9+
from autograder.models.pipeline_execution import PipelineExecution
510

611

712
class Exporter(ABC):
@@ -10,6 +15,13 @@ class Exporter(ABC):
1015
1116
Any system that receives grading results (Upstash Redis, GitHub Classroom, etc.)
1217
must implement this interface to be usable with the pipeline's ExporterStep.
18+
19+
Subclasses that need access to the full pipeline execution (result tree,
20+
focus data, execution time, etc.) should override
21+
:meth:`export_with_context` instead of — or in addition to — :meth:`export`.
22+
The default implementation of :meth:`export_with_context` extracts
23+
``user_id``, ``score``, and ``feedback`` from the execution and delegates
24+
to :meth:`export`, so existing exporters require no changes.
1325
"""
1426

1527
@abstractmethod
@@ -23,4 +35,21 @@ def export(self, user_id: str, score: float, feedback: Optional[str] = None) ->
2335
feedback: Optional feedback text. Exporters may ignore this
2436
if they only handle scores.
2537
"""
26-
pass
38+
39+
def export_with_context(self, pipeline_exec: "PipelineExecution") -> None:
40+
"""
41+
Export a grading result with access to the full pipeline execution.
42+
43+
Override this in exporters that require richer data (e.g., result
44+
tree, focus, execution time). The default implementation extracts
45+
``user_id``, ``score``, and ``feedback`` from *pipeline_exec* and
46+
delegates to :meth:`export`.
47+
48+
Args:
49+
pipeline_exec: The completed :class:`~autograder.models.pipeline_execution.PipelineExecution`
50+
containing all step results.
51+
"""
52+
user_id = pipeline_exec.submission.user_id
53+
score = pipeline_exec.get_grade_step_result().final_score
54+
feedback: Optional[str] = pipeline_exec.get_feedback()
55+
self.export(user_id, score, feedback)

‎autograder/steps/export_step.py‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
import logging
2-
from typing import Optional
32

43
from autograder.models.abstract.exporter import Exporter
54
from autograder.models.abstract.step import Step
@@ -30,13 +29,13 @@ def _execute(self, pipeline_exec: PipelineExecution) -> PipelineExecution:
3029
Returns:
3130
PipelineExecution with an added StepResult indicating success or failure of the export operation
3231
"""
33-
# Extract external_user_id, score and optional feedback
3432
external_user_id = pipeline_exec.submission.user_id
35-
score = pipeline_exec.get_grade_step_result().final_score
36-
feedback: Optional[str] = pipeline_exec.get_feedback()
37-
38-
logger.info("Exporting result: external_user_id=%s, score=%.2f", external_user_id, score)
39-
self._exporter_service.export(external_user_id, score, feedback)
33+
try:
34+
score = pipeline_exec.get_grade_step_result().final_score
35+
logger.info("Exporting result: external_user_id=%s, score=%.2f", external_user_id, score)
36+
except (ValueError, AttributeError):
37+
logger.info("Exporting result: external_user_id=%s (score unavailable)", external_user_id)
38+
self._exporter_service.export_with_context(pipeline_exec)
4039
logger.info("Result exported successfully: external_user_id=%s", external_user_id)
4140

4241
# Return success result

‎docs/API.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -853,6 +853,102 @@ GET /api/v1/ready
853853

854854
---
855855

856+
## External Mode Integration
857+
858+
This section describes how the Autograder GitHub Action in **external mode** interacts with the API. The action makes exactly two authenticated calls per grading run.
859+
860+
### Integration sequence
861+
862+
```
863+
1. GET /api/v1/configs/id/{grading_config_id} # fetch grading config
864+
2. POST /api/v1/submissions/external-results # submit result (or failure)
865+
```
866+
867+
Both endpoints require the `Authorization: Bearer <token>` header where the token matches `AUTOGRADER_INTEGRATION_TOKEN` on the server.
868+
869+
### Config fetch
870+
871+
```http
872+
GET /api/v1/configs/id/{grading_config_id}
873+
Authorization: Bearer <token>
874+
```
875+
876+
Returns the full grading configuration object (see [Get Grading Configuration by Internal ID](#get-grading-configuration-by-internal-id)). The action uses the following fields from the response:
877+
878+
| Field | Used for |
879+
|-------|----------|
880+
| `id` | Sent back as `grading_config_id` in the result payload |
881+
| `template_name` | Selects the grading template |
882+
| `criteria_config` | Grading rubric |
883+
| `languages` | Validates `submission-language`; defaults to `languages[0]` if omitted |
884+
| `include_feedback` | Whether to run the feedback step |
885+
| `setup_config` | Pre-flight setup commands and required files |
886+
| `feedback_config` | Reporter configuration |
887+
888+
### Result submission (success)
889+
890+
```http
891+
POST /api/v1/submissions/external-results
892+
Authorization: Bearer <token>
893+
Content-Type: application/json
894+
895+
{
896+
"grading_config_id": 42,
897+
"external_user_id": "github-actor-login",
898+
"username": "github-actor-login",
899+
"language": "python",
900+
"status": "completed",
901+
"final_score": 85.5,
902+
"feedback": "## Grade: 85.5/100\n...",
903+
"result_tree": { ... },
904+
"focus": { ... },
905+
"pipeline_execution": { ... },
906+
"execution_time_ms": 3200,
907+
"error_message": null,
908+
"submission_metadata": {
909+
"repository": "org/repo",
910+
"commit_sha": "abc123",
911+
"run_id": "999",
912+
"actor": "student1",
913+
"ref": "refs/heads/main"
914+
}
915+
}
916+
```
917+
918+
### Result submission (failure)
919+
920+
When grading fails, the action posts a failure payload before exiting non-zero:
921+
922+
```json
923+
{
924+
"grading_config_id": 42,
925+
"external_user_id": "github-actor-login",
926+
"username": "github-actor-login",
927+
"language": "python",
928+
"status": "failed",
929+
"final_score": 0.0,
930+
"feedback": null,
931+
"result_tree": null,
932+
"focus": null,
933+
"pipeline_execution": null,
934+
"execution_time_ms": 1500,
935+
"error_message": "Sandbox timed out after 30 s",
936+
"submission_metadata": { ... }
937+
}
938+
```
939+
940+
### Error handling
941+
942+
| Scenario | Action behaviour |
943+
|----------|-----------------|
944+
| `401` on config fetch | Exits non-zero; no result posted |
945+
| `404` on config fetch | Exits non-zero; no result posted |
946+
| 5xx / network error on config fetch | Retries with exponential back-off; then exits non-zero |
947+
| Grading raises exception | Posts failure payload, then exits non-zero |
948+
| `401` / `4xx` on result submission | Exits non-zero (failure recorded in Action log) |
949+
950+
---
951+
856952
## Error Responses
857953

858954
All endpoints use FastAPI's standard error response format:

‎docs/github_action/README.md‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,33 @@ This module packages the Autograder as a Docker-based GitHub Action so repositor
44

55
It is primarily designed for GitHub Classroom style workflows, but can be used by any repository that follows the expected directory contract.
66

7-
## What it does
7+
## Execution modes
88

9-
When a workflow runs:
9+
| Mode | How config is loaded | When to use |
10+
|------|---------------------|-------------|
11+
| `repo` (default) | From config files checked out inside `submission/.github/autograder/` | Repository-based setups where grading config lives in the student repo |
12+
| `external` | Fetched from the Autograder Cloud API at runtime | Centralised setups where a single Autograder Cloud instance manages all assignments |
13+
14+
## What it does (repo mode)
15+
16+
When a workflow runs in `repo` mode:
1017

1118
1. Reads student files from `submission/`.
1219
2. Reads grading config from `submission/.github/autograder/`.
1320
3. Builds and executes the Autograder pipeline.
1421
4. Updates the GitHub check run with the final score.
1522
5. Optionally commits `relatorio.md` with feedback.
1623

24+
## What it does (external mode)
25+
26+
When a workflow runs in `external` mode:
27+
28+
1. Fetches the grading configuration from the Autograder Cloud (`GET /api/v1/configs/id/{id}`).
29+
2. Reads student files from `submission/`.
30+
3. Builds and executes the Autograder pipeline.
31+
4. Posts the grading result back to the Autograder Cloud (`POST /api/v1/submissions/external-results`).
32+
5. If grading fails for any reason, posts a `failed` status payload before exiting non-zero.
33+
1734
## Internal architecture
1835

1936
| Component | Responsibility |
@@ -24,6 +41,8 @@ When a workflow runs:
2441
| [`github_action/main.py`](https://github.com/webtech-network/autograder/blob/main/github_action/main.py) | Parses arguments, validates runtime options, reads submission files, and starts grading. |
2542
| [`github_action/github_action_service.py`](https://github.com/webtech-network/autograder/blob/main/github_action/github_action_service.py) | Builds pipeline from JSON configs and handles GitHub export operations. |
2643
| [`github_action/github_classroom_exporter.py`](https://github.com/webtech-network/autograder/blob/main/github_action/github_classroom_exporter.py) | Export adapter that reports score and feedback through `GithubActionService`. |
44+
| [`github_action/cloud_client.py`](https://github.com/webtech-network/autograder/blob/main/github_action/cloud_client.py) | HTTP client for the Autograder Cloud API (config fetch + result submission). Uses exponential back-off on 5xx/network errors; raises `CloudClientError` on 4xx. |
45+
| [`github_action/cloud_exporter.py`](https://github.com/webtech-network/autograder/blob/main/github_action/cloud_exporter.py) | Export adapter for external mode. Builds the `ExternalResultCreate` payload and calls `CloudClient`. Provides `submit_failure()` for the failure path. |
2746

2847
## Repository contract required by the Action
2948

@@ -47,6 +66,8 @@ Inside that checkout, the Action expects:
4766
- `feedback-type: ai` requires `openai-key`.
4867
- The current notification logic looks for a check run named `grading`.
4968
- `include-feedback: "true"` enables feedback generation and may update `relatorio.md`.
69+
- In `external` mode, `include_feedback` is taken from the cloud config, **not** from the `include-feedback` action input.
70+
- In `external` mode, `submission-language` is validated against the `languages` list in the cloud config. Omitting it defaults to the first language in that list.
5071

5172
## Reference repository
5273

@@ -55,4 +76,5 @@ Use **[`webtech-network/demo-autograder`](https://github.com/webtech-network/dem
5576
## Next
5677

5778
- Configuration details: [configuration.md](configuration.md)
79+
- External mode deep-dive: [external-mode.md](external-mode.md)
5880
- Demo walkthrough: [demo-autograder.md](demo-autograder.md)

‎docs/github_action/configuration.md‎

Lines changed: 60 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ This page describes how to configure `webtech-network/autograder@main` in your w
44

55
## Recommended workflow skeleton
66

7+
### Repo mode (default)
8+
79
```yaml
810
name: Autograder
911
on:
@@ -33,19 +35,55 @@ jobs:
3335
openai-key: ${{ secrets.ENGINE }}
3436
```
3537
38+
### External mode
39+
40+
```yaml
41+
name: Autograder (External)
42+
on:
43+
push:
44+
branches: [main]
45+
workflow_dispatch:
46+
47+
jobs:
48+
grading:
49+
runs-on: ubuntu-latest
50+
if: github.actor != 'github-classroom[bot]'
51+
steps:
52+
- name: Checkout repository
53+
uses: actions/checkout@v4
54+
with:
55+
path: submission
56+
57+
- name: Run Autograder
58+
uses: webtech-network/autograder@main
59+
with:
60+
execution-mode: "external"
61+
grading-config-id: "42" # internal DB id
62+
autograder-cloud-url: ${{ secrets.AUTOGRADER_CLOUD_URL }}
63+
autograder-cloud-token: ${{ secrets.AUTOGRADER_CLOUD_TOKEN }}
64+
submission-language: "python" # optional
65+
feedback-type: "default"
66+
template-preset: "input_output" # required by the parser; value is ignored in external mode
67+
```
68+
3669
## Inputs
3770
3871
From [`action.yml`](https://github.com/webtech-network/autograder/blob/main/action.yml):
3972

4073
| Input | Required | Default | Notes |
4174
|---|---:|---|---|
4275
| `github-token` | no | `${{ github.token }}` | Used for GitHub API operations (repo/check run access). |
43-
| `template-preset` | yes | - | Template key used by the grading pipeline. Built-ins include `webdev`, `api`, and `input_output`. |
76+
| `template-preset` | yes | - | Template key used by the grading pipeline. Built-ins include `webdev`, `api`, and `input_output`. In external mode the pipeline reads the template from the cloud config, but this field is still required by the parser. |
4477
| `feedback-type` | yes | - | `default` or `ai`. |
4578
| `custom-template` | no | - | Currently not usable because `template-preset: custom` is rejected by `github_action/main.py`. |
4679
| `openai-key` | no | - | Required only when `feedback-type` is `ai`. |
4780
| `app-token` | no | - | Optional token for repository access; if omitted, runtime falls back to `github-token`. |
48-
| `include-feedback` | no | `"false"` | Must be `"true"` or `"false"` string. Controls whether feedback step is included. |
81+
| `include-feedback` | no | `"false"` | Must be `"true"` or `"false"` string. Ignored in external mode (value is taken from the cloud config). |
82+
| `execution-mode` | no | `"repo"` | `"repo"` or `"external"`. Determines where grading configuration is loaded from. |
83+
| `grading-config-id` | no | - | Internal DB id of the grading config on the Autograder Cloud. Required when `execution-mode` is `external`. |
84+
| `autograder-cloud-url` | no | - | Base URL of the Autograder Cloud instance (e.g. `https://cloud.example.com`). Required when `execution-mode` is `external`. |
85+
| `autograder-cloud-token` | no | - | Integration token for the Autograder Cloud API. Required when `execution-mode` is `external`. Should be stored as a repository secret. |
86+
| `submission-language` | no | - | Language of the student submission (e.g. `python`, `java`). Validated against the cloud config's `languages` list. Defaults to the first language in the list when omitted. Only used in `external` mode. |
4987

5088
## Outputs
5189

@@ -72,24 +110,41 @@ It also reads student files recursively from `submission/`, excluding `.git` and
72110

73111
## Secrets and permissions
74112

75-
### Typical secrets
113+
### Repo mode secrets
76114

77115
- `ENGINE` (or another secret mapped to `openai-key`) when using AI feedback.
78116
- Optional token secret mapped to `app-token` if you need separate credentials.
79117

118+
### External mode secrets
119+
120+
| Secret | Maps to input | Description |
121+
|--------|--------------|-------------|
122+
| `AUTOGRADER_CLOUD_URL` | `autograder-cloud-url` | Base URL of the Autograder Cloud instance |
123+
| `AUTOGRADER_CLOUD_TOKEN` | `autograder-cloud-token` | Integration token (set `AUTOGRADER_INTEGRATION_TOKEN` on the server) |
124+
125+
> Generate a strong token on the server: `openssl rand -hex 32`
126+
80127
### Permissions
81128

82-
`permissions: write-all` is commonly used because the Action may:
129+
`permissions: write-all` is required in **repo mode** because the Action may:
83130

84131
- update the workflow check run with score summary;
85132
- commit `relatorio.md` when feedback is enabled.
86133

134+
In **external mode**, no GitHub repository write operations are performed, so elevated permissions are not required.
135+
87136
## Troubleshooting
88137

89138
### `FileNotFoundError: criteria.json file not found`
90139

91140
- Ensure your workflow checks out repository to `path: submission`.
92141
- Ensure config is at `submission/.github/autograder/criteria.json`.
142+
- This error only occurs in **repo mode**; in external mode, config is fetched from the cloud.
143+
144+
### `grading-config-id`, `autograder-cloud-url`, or `autograder-cloud-token` missing
145+
146+
- All three are required when `execution-mode` is `external`.
147+
- The entrypoint validates these before starting the pipeline and exits non-zero if any are absent.
93148

94149
### `OpenAI API key is required for AI feedback mode`
95150

@@ -107,4 +162,5 @@ It also reads student files recursively from `submission/`, excluding `.git` and
107162
## See also
108163

109164
- Module internals: [README.md](README.md)
165+
- External mode deep-dive: [external-mode.md](external-mode.md)
110166
- Demo repository walkthrough: [demo-autograder.md](demo-autograder.md)

0 commit comments

Comments
 (0)