Skip to content

Commit 955671e

Browse files
jmgasperjmgasper
committed
Documentation on ECS runner lifetime
1 parent af07d25 commit 955671e

2 files changed

Lines changed: 276 additions & 0 deletions

File tree

‎ECS_RUNNER_LIFETIME.md‎

Lines changed: 275 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,275 @@
1+
# ECS Runner Lifetime
2+
3+
This document describes how `marathon-match-api-v6` starts ECS scorer runners, how to monitor them while they are active, and how to stop them when the work is no longer needed.
4+
5+
The runner task is the `ecs-runner` Fargate container built from `ecs-runner/Dockerfile`. The API never launches an image directly; it launches the configured ECS task definition revision from each Marathon Match config.
6+
7+
## Required launch configuration
8+
9+
Before any runner can start, the API service needs these ECS/runtime values:
10+
11+
- `ECS_CLUSTER`
12+
- `ECS_SUBNETS`
13+
- `ECS_SECURITY_GROUPS`
14+
- `ECS_CONTAINER_NAME`
15+
- `AWS_REGION` (defaults to `us-east-1`)
16+
- `MARATHON_MATCH_API_URL`
17+
- `REVIEW_API_URL`
18+
- `REVIEW_TYPE_ID`
19+
- Auth0 M2M settings used by the API and forwarded to the trusted parent runner for long-running token refresh
20+
21+
Each active challenge config also needs:
22+
23+
- `active: true`
24+
- compiled tester with `compilationStatus = SUCCESS`
25+
- `taskDefinitionName`
26+
- `taskDefinitionVersion`
27+
- phase configs for `EXAMPLE`, `PROVISIONAL`, and/or `SYSTEM`
28+
29+
## Start paths
30+
31+
All start paths converge on `EcsService.launchScorerTask(...)`.
32+
33+
### Live submission scoring
34+
35+
1. A `marathonmatch.submission.received` Kafka message arrives with `challengeId`, `submissionId`, and usually `memberId`.
36+
2. `MarathonMatchSubmissionHandler` loads the challenge's Marathon Match config.
37+
3. The handler skips the submission if the config is inactive, the challenge has no open phase, or no stored phase config matches the open challenge phase.
38+
4. For every matching phase config, the handler calls `EcsService.launchScorerTask(...)`.
39+
40+
This is the normal path for `EXAMPLE` and `PROVISIONAL` scoring during the Submission phase. If both phase configs map to the same open challenge phase, the API intentionally launches one runner per matching config.
41+
42+
### Validation submission scoring
43+
44+
`POST /v6/marathon-match/challenge/:challengeId/test-submission` uploads a validation submission through the configured Submission API and then launches one ECS runner for the requested `configType` (default `PROVISIONAL`). The response includes `submissionId`, `taskArn`, `taskId`, and usually `cloudWatchLogsConsoleUrl`.
45+
46+
### Manual latest-submission rerun
47+
48+
`POST /v6/marathon-match/challenge/:challengeId/rerun` finds the latest submission per member and launches scorer tasks for the phase config that matches the currently open challenge phase. Use this after changing tester configuration or when current latest submissions need to be recalculated.
49+
50+
### SYSTEM review scoring
51+
52+
SYSTEM scoring starts from review orchestration:
53+
54+
1. Autopilot creates or finds a pending Review API review for a latest submission.
55+
2. Autopilot calls `POST /v6/marathon-match/internal/system-score` with `reviewId`, `submissionId`, and `challengeId`.
56+
3. `ScoringResultService.triggerSystemScore(...)` loads the config, launches the SYSTEM runner, and schedules a pg-boss timeout guard using `systemTestTimeout`.
57+
58+
`POST /v6/marathon-match/challenge/:challengeId/rerun/system` restarts existing non-cancelled SYSTEM review records through the same SYSTEM launch path.
59+
60+
## What launch does
61+
62+
`EcsService.launchScorerTask(...)` performs these steps before and during `RunTask`:
63+
64+
1. Reads the configured ECS cluster, container name, network settings, API URL, review type, task definition family, and revision.
65+
2. Lists active `PENDING` and `RUNNING` tasks for the task family, then describes them and inspects container override environment values.
66+
3. Stops older active tasks for the same `challengeId` and `memberId` when a newer submission for that member is being launched.
67+
4. Reuses an existing active task instead of launching a duplicate for the same challenge, submission, and phase config type.
68+
5. Enforces `ECS_SCORER_MAX_CONCURRENT_TASKS` before creating more work. The default cap is `20`.
69+
6. Fetches an M2M token and injects runner environment variables into the configured ECS container.
70+
7. Calls ECS `RunTask` with `launchType: FARGATE`, `awsvpc` networking, and `assignPublicIp: DISABLED`.
71+
8. Resolves the task definition's `awslogs-group` and `awslogs-stream-prefix`, then builds the deterministic stream name as `<prefix>/<containerName>/<taskId>`.
72+
9. Logs and persists a `submissionRunnerLog` mapping with the submission, task, cluster, container, task definition, CloudWatch log metadata, and phase config type.
73+
74+
The runner receives these main environment values:
75+
76+
- `TESTER_CONFIG_ID`
77+
- `SUBMISSION_ID`
78+
- `ACCESS_TOKEN`
79+
- `MARATHON_MATCH_API_URL`
80+
- `REVIEW_TYPE_ID`
81+
- `TEST_PHASE`
82+
- `PHASE_CONFIG_TYPE`
83+
- `PHASE_START_SEED`
84+
- `PHASE_NUMBER_OF_TESTS`
85+
- `REVIEW_ID` for SYSTEM scoring
86+
- `MEMBER_ID` when known
87+
88+
## Runner execution
89+
90+
Inside the task, the trusted Java parent runner:
91+
92+
1. Fetches the challenge config, tester JAR, and tester metadata from `marathon-match-api-v6`.
93+
2. Downloads the submission from the configured Submission API URL.
94+
3. Posts an initial progress callback for `PROVISIONAL` and `SYSTEM` runs.
95+
4. Starts the tester in an isolated child JVM. Submitted solution commands run as the separate `scorer` user with scrubbed environment, restricted filesystem access, and no outbound INET/INET6 socket creation.
96+
5. Uploads public/private artifacts back through Submission API.
97+
6. Posts the final scoring callback to `POST /v6/marathon-match/internal/scoring-results`.
98+
7. Exits with code `0` only after the final callback succeeds.
99+
100+
On failure, the runner writes failure artifacts, uploads them when possible, posts best-effort failed progress for tracked phases, logs `runner.failure`, and exits non-zero.
101+
102+
## Monitoring
103+
104+
### Launch mapping
105+
106+
Every launch logs `Launched ECS scorer task` and `Submission to ECS runner log mapping`. The mapping is also stored in `marathon_match.submissionRunnerLog`.
107+
108+
The mapping contains:
109+
110+
- `submissionId`
111+
- `challengeId`
112+
- `taskArn`
113+
- `taskId`
114+
- `cluster`
115+
- `containerName`
116+
- `taskDefinition`
117+
- `phaseConfigType`
118+
- `logGroup`
119+
- `logStreamName`
120+
- `cloudWatchLogsConsoleUrl`
121+
122+
### Runner logs endpoint
123+
124+
Use the API endpoint when you have a submission ID:
125+
126+
```bash
127+
curl -H "Authorization: Bearer $TOKEN" \
128+
"$MARATHON_MATCH_API_URL/submissions/$SUBMISSION_ID/runner-logs"
129+
```
130+
131+
Add `taskArn` when a submission has multiple runner launches:
132+
133+
```bash
134+
curl -H "Authorization: Bearer $TOKEN" \
135+
"$MARATHON_MATCH_API_URL/submissions/$SUBMISSION_ID/runner-logs?taskArn=$TASK_ARN&limit=200"
136+
```
137+
138+
The endpoint returns all known mappings for the submission, the selected mapping, CloudWatch log events, and CloudWatch pagination tokens.
139+
140+
### CloudWatch logs
141+
142+
If you have `logGroup` and `logStreamName` from the mapping:
143+
144+
```bash
145+
aws logs tail "$LOG_GROUP" \
146+
--region "$AWS_REGION" \
147+
--log-stream-names "$LOG_STREAM_NAME" \
148+
--follow
149+
```
150+
151+
Useful runner log markers include:
152+
153+
- `bootstrap`
154+
- `api.fetch-config`
155+
- `api.fetch-tester-jar`
156+
- `api.download-submission`
157+
- `tester.isolated`
158+
- `tester.invoke`
159+
- `api.progress`
160+
- `artifacts.upload`
161+
- `api.callback`
162+
- `runner.failure`
163+
- `cleanup`
164+
- `exit`
165+
166+
Success usually includes `Scoring callback completed successfully`, a runner result payload, and `Exiting runner with code 0`.
167+
168+
### ECS task status
169+
170+
Use ECS when you need runtime/container status:
171+
172+
```bash
173+
aws ecs describe-tasks \
174+
--region "$AWS_REGION" \
175+
--cluster "$ECS_CLUSTER" \
176+
--tasks "$TASK_ARN" \
177+
--query 'tasks[0].{lastStatus:lastStatus,desiredStatus:desiredStatus,stoppedReason:stoppedReason,containers:containers[*].{name:name,lastStatus:lastStatus,exitCode:exitCode,reason:reason}}'
178+
```
179+
180+
`lastStatus = STOPPED` means ECS considers the task terminal. Check the container `exitCode` and `reason` to distinguish normal completion from runner failures, OOM stops, or manual stops.
181+
182+
### Review progress metadata
183+
184+
For `PROVISIONAL` and `SYSTEM`, the runner posts progress to `POST /v6/marathon-match/internal/scoring-progress`. Review summation metadata is updated with:
185+
186+
- `testProcess`: `provisional` or `system`
187+
- `testProgress`: value from `0` to `1`
188+
- `testStatus`: `IN PROGRESS`, `SUCCESS`, or `FAILED`
189+
- `timed_out: true` for SYSTEM timeout failures
190+
191+
Treat `IN PROGRESS` summations as unavailable, even when they carry a placeholder score.
192+
193+
## Stopping or killing runners
194+
195+
### Stop new work first
196+
197+
To stop future Marathon Match launches for a challenge, set the config inactive:
198+
199+
```bash
200+
curl -X PUT "$MARATHON_MATCH_API_URL/challenge/$CHALLENGE_ID" \
201+
-H "Authorization: Bearer $TOKEN" \
202+
-H "Content-Type: application/json" \
203+
-d '{"active":false}'
204+
```
205+
206+
This prevents new scoring dispatch for that config. It does not stop tasks that are already running.
207+
208+
### Stop one active ECS task
209+
210+
There is no public HTTP endpoint that wraps `EcsService.stopTask(...)`. For an operator-initiated kill, use ECS directly with the task ARN from the runner mapping:
211+
212+
```bash
213+
aws ecs stop-task \
214+
--region "$AWS_REGION" \
215+
--cluster "$ECS_CLUSTER" \
216+
--task "$TASK_ARN" \
217+
--reason "Marathon Match runner no longer needed"
218+
```
219+
220+
Then verify the task reached `STOPPED`:
221+
222+
```bash
223+
aws ecs describe-tasks \
224+
--region "$AWS_REGION" \
225+
--cluster "$ECS_CLUSTER" \
226+
--tasks "$TASK_ARN" \
227+
--query 'tasks[0].{lastStatus:lastStatus,desiredStatus:desiredStatus,stoppedReason:stoppedReason,containers:containers[*].{name:name,exitCode:exitCode,reason:reason}}'
228+
```
229+
230+
Manual ECS stops can interrupt the Java process before it posts final scoring results. After a manual stop, check Review summation metadata and decide whether to rerun, leave the result unavailable, or apply an operator-side failure workflow.
231+
232+
### Automatic same-member cancellation
233+
234+
For submission-phase launches that include `memberId`, the API stops older active scorer tasks for the same challenge/member before launching the newer submission. The ECS stop reason is:
235+
236+
`Superseded by newer Marathon Match submission <submissionId> for challenge <challengeId>.`
237+
238+
This is the normal cleanup path for obsolete in-flight work when a competitor submits again.
239+
240+
### SYSTEM timeout stop
241+
242+
Every SYSTEM launch schedules a `system-test-timeout` pg-boss job unless `DISABLE_PG_BOSS=true` or timeout scheduling is unavailable. The timeout uses `systemTestTimeout` from the challenge config and defaults to 24 hours.
243+
244+
When the timeout job runs, it:
245+
246+
1. Describes the ECS task.
247+
2. Skips work if the task is already stopped.
248+
3. Checks whether SYSTEM scoring is already complete.
249+
4. Calls `EcsService.stopTask(...)` if the task is still active and scoring is incomplete.
250+
5. Writes a failed SYSTEM result with score `-1` and metadata containing `timed_out: true`, `timeoutMs`, `taskArn`, and a timeout message.
251+
252+
### Stop all active tasks for a task family
253+
254+
Use this only after confirming the task family is dedicated to Marathon Match scoring in the target environment:
255+
256+
```bash
257+
aws ecs list-tasks \
258+
--region "$AWS_REGION" \
259+
--cluster "$ECS_CLUSTER" \
260+
--family "$TASK_FAMILY" \
261+
--desired-status RUNNING \
262+
--query 'taskArns' \
263+
--output text
264+
```
265+
266+
Stop selected ARNs one by one with `aws ecs stop-task`. Repeat the list command with `--desired-status PENDING` if you also need to cancel queued tasks that have not reached `RUNNING`.
267+
268+
## Practical lifecycle checklist
269+
270+
1. Confirm the challenge config is active, tester compilation succeeded, and task definition values point to the intended ECS revision.
271+
2. Start work through the natural trigger: Kafka submission event, validation upload, rerun endpoint, or SYSTEM scoring endpoint.
272+
3. Capture `taskArn`, `taskId`, `logGroup`, and `logStreamName` from the API response, API logs, or `GET /submissions/:submissionId/runner-logs`.
273+
4. Monitor CloudWatch logs, ECS task status, and Review summation progress metadata.
274+
5. When work is obsolete, deactivate the config if needed, then stop the selected ECS task ARN.
275+
6. After a manual stop, verify ECS `STOPPED` status and clean up the scoring state by rerunning or marking the result through the appropriate operator workflow.

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Swagger UI:
1515
## Documentation
1616

1717
- [Marathon Match Setup](docs/marathon-match-setup.md) - for challenge administrators
18+
- [ECS Runner Lifetime](ECS_RUNNER_LIFETIME.md) - how scorer tasks are launched, monitored, and stopped
1819
- [Submission Phase Scoring](docs/submission-phase-scoring.md) - for DevOps
1920
- [Review Phase Scoring](docs/review-phase-scoring.md) - for DevOps
2021
- [Full Marathon Match Test Script](docs/full-marathon-match-test.md) - for end-to-end smoke tests with production baseline fixtures

0 commit comments

Comments
 (0)