Skip to content

Commit af07d25

Browse files
committed
Add restart endpoint for system tests.
1 parent 36a5b42 commit af07d25

12 files changed

Lines changed: 781 additions & 15 deletions

docs/Marathon Match Setup and Management.md

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -285,22 +285,35 @@ Use this flow when the tester source changes after the challenge is active.
285285
3. Create the new version with the updated Java source.
286286
4. Wait until compilation reaches `SUCCESS`.
287287
5. Save the scorer config so the challenge points at the new tester ID.
288-
6. Trigger a rerun of latest submissions using the `Rerun scores` button on the UI.
289-
7. Monitor the rerun ECS tasks, runner logs, artifacts, and review scores.
288+
6. Trigger a rerun of latest submissions for the open Submission/Provisional phase using the Work app rerun control or `POST /v6/marathon-match/challenge/:challengeId/rerun`.
289+
7. If Review/System scores also need to be regenerated, call `POST /v6/marathon-match/challenge/:challengeId/rerun/system`.
290+
8. Monitor the rerun ECS tasks, runner logs, artifacts, and review scores.
290291

291292

292-
Rerun behavior:
293+
Submission/Provisional rerun behavior:
293294

294295
- Requires the challenge to be `ACTIVE`.
295296
- Requires at least one open challenge phase.
296297
- Requires the scorer config to be active.
297298
- Requires the selected tester to be compiled successfully.
298-
- Requires a `PROVISIONAL` phase config.
299+
- Requires a phase config that matches the currently open challenge phase.
299300
- Fetches `isLatest=true` submissions from Submission API.
300301
- Queues one ECS task for the latest submission per member.
301-
- Uses the current `PROVISIONAL` seeds and test count.
302+
- Uses the current open phase's scorer seeds and test count. During Submission this is normally the `PROVISIONAL` config.
302303

303-
The response includes the queued submission IDs and ECS task IDs or per-submission errors. Because the standard rerun endpoint is provisional-oriented, coordinate separately if a late tester change also needs Review/System scores regenerated.
304+
The response includes the queued submission IDs and ECS task IDs or per-submission errors.
305+
306+
System rerun behavior:
307+
308+
- Requires the challenge to be `ACTIVE`.
309+
- Requires the scorer config to be active.
310+
- Requires the selected tester to be compiled successfully.
311+
- Requires a `SYSTEM` phase config.
312+
- Fetches existing non-cancelled Review API reviews for the challenge that match the configured Marathon Match review scorecard.
313+
- Queues one SYSTEM scorer task per matching review and passes the existing `reviewId` to the runner.
314+
- Uses the current `SYSTEM` seeds, test count, per-test timeout, and `systemTestTimeout`.
315+
316+
Use the SYSTEM rerun endpoint after lowering timings or changing a tester while Review is open. It restarts the existing review records instead of creating duplicate reviews.
304317

305318
## Monitoring Submissions
306319

docs/marathon-match-setup.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,13 @@ To switch an active challenge to a newer tester:
160160
3. Update the challenge config with `PUT /v6/marathon-match/challenge/:challengeId` and the new `testerId`
161161
4. Trigger a rescore of current competitors with `POST /v6/marathon-match/challenge/:challengeId/rerun`
162162

163-
The rerun endpoint selects `isLatest` submissions for the challenge in received order and launches ECS scorer tasks for them in parallel. This does not happen automatically when `testerId` changes, so the rerun call is the operational step that recalculates scores.
163+
The standard rerun endpoint selects `isLatest` submissions for the challenge in received order and launches ECS scorer tasks for them in parallel using the scorer config for the currently open challenge phase. This does not happen automatically when `testerId` changes, so the rerun call is the operational step that recalculates Submission/Provisional scores.
164+
165+
To restart Review/System tests, call:
166+
167+
`POST /v6/marathon-match/challenge/:challengeId/rerun/system`
168+
169+
The SYSTEM rerun endpoint reuses existing non-cancelled Review API review records that match the challenge's configured Marathon Match scorecard and dispatches each one through SYSTEM scoring with its existing `reviewId`.
164170

165171
Reruns can be requested by admins, M2M tokens with `update:marathon-match`, or the `Copilot` resource assigned to the challenge.
166172

docs/marathon-processor-specification-and-scoring-terminology.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ Compile and test timeouts are configured per Marathon Match challenge.
110110
initializer checks performed after `javac`.
111111
- `testTimeout` controls per-seed tester execution timeout.
112112
- `systemTestTimeout` controls the total SYSTEM scoring timeout per submission. It defaults to 24 hours and causes the API to stop a still-active ECS runner and write a failed SYSTEM summation with `metadata.timed_out = true`.
113+
- `POST /v6/marathon-match/challenge/:challengeId/rerun/system` restarts existing non-cancelled SYSTEM reviews with the current `testTimeout`, `systemTestTimeout`, SYSTEM seed, and SYSTEM test count settings.
113114

114115
The default values supplied by `marathon-match-api-v6` are environment-configurable. If not overridden in service configuration, the API defaults are:
115116

docs/review-phase-scoring.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ When the Review phase opens for a Marathon Match challenge, `autopilot-v6` route
1010

1111
Manual phase changes are supported through the same path. If Review is opened directly in challenge-api, autopilot's challenge-update handling replays the phase-open work; startup recovery also replays already-open Review phases so missed update events still create and dispatch SYSTEM reviews.
1212

13+
Existing SYSTEM reviews can be restarted through `POST /v6/marathon-match/challenge/:challengeId/rerun/system`. This endpoint is intended for operator reruns after lowering timing settings or changing the tester while Review is open. It finds non-cancelled Review API reviews matching the challenge's configured Marathon Match scorecard, then dispatches SYSTEM scorer tasks with each existing `reviewId`.
14+
1315
## Flow
1416

1517
```mermaid

docs/submission-phase-scoring.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,12 @@ Use:
118118

119119
`POST /v6/marathon-match/challenge/:challengeId/rerun`
120120

121-
This endpoint selects the latest submission for each member in received order and launches ECS scorer tasks in parallel using the challenge's `PROVISIONAL` phase config. Use it after changing the tester or when current latest submissions need to be rescored without waiting for new submission events.
121+
This endpoint selects the latest submission for each member in received order and launches ECS scorer tasks in parallel using the scorer config that matches the currently open challenge phase. During Submission this is normally the challenge's `PROVISIONAL` phase config. Use it after changing the tester or when current latest submissions need to be rescored without waiting for new submission events.
122+
123+
For Review/System scoring reruns, use:
124+
125+
`POST /v6/marathon-match/challenge/:challengeId/rerun/system`
126+
127+
The SYSTEM rerun endpoint restarts existing non-cancelled Review API reviews that match the challenge's configured Marathon Match review scorecard and passes each existing `reviewId` into the SYSTEM scorer.
122128

123129
Rerun access is limited to admins, M2M tokens with `update:marathon-match`, and the `Copilot` resource assigned to the challenge.

src/api/marathon-match-config/marathon-match-config.controller.ts

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ import {
3030
MarathonMatchDefaultsResponseDto,
3131
RerunResponseDto,
3232
SearchMarathonMatchConfigQueryDto,
33+
SystemRerunResponseDto,
3334
UpdateMarathonMatchConfigDto,
3435
} from 'src/dto/marathon-match-config.dto';
3536
import { PaginationHeaderInterceptor } from 'src/interceptors/PaginationHeaderInterceptor';
@@ -148,6 +149,50 @@ export class MarathonMatchConfigController {
148149
);
149150
}
150151

152+
/**
153+
* Reruns existing SYSTEM review tests for a marathon match configuration.
154+
* @param challengeId Challenge ID.
155+
* @param user Authenticated user for audit context.
156+
* @returns Accepted SYSTEM rerun dispatch summary.
157+
*/
158+
@Post('/:challengeId/rerun/system')
159+
@Roles(UserRole.Admin, UserRole.Copilot, UserRole.User)
160+
@Scopes(Scope.UpdateMarathonMatch)
161+
@UseGuards(ChallengeCopilotResourceGuard)
162+
@ApiOperation({
163+
summary: 'Rerun existing SYSTEM tests for a Marathon Match challenge',
164+
description:
165+
'Roles: Admin or challenge Copilot resource | Scopes: update:marathon-match',
166+
})
167+
@ApiParam({
168+
name: 'challengeId',
169+
description: 'The challenge ID for the marathon match SYSTEM rerun request',
170+
example: '30000123',
171+
})
172+
@HttpCode(HttpStatus.ACCEPTED)
173+
@ApiResponse({
174+
status: 202,
175+
description:
176+
'Existing SYSTEM reviews queued for asynchronous rerun dispatch.',
177+
type: SystemRerunResponseDto,
178+
})
179+
@ApiResponse({
180+
status: 400,
181+
description:
182+
'Challenge/config inactive, tester is not compiled successfully, or SYSTEM phase config is missing.',
183+
})
184+
@ApiResponse({ status: 403, description: 'Forbidden.' })
185+
@ApiResponse({ status: 404, description: 'Marathon match config not found.' })
186+
async rerunSystemTests(
187+
@Param('challengeId') challengeId: string,
188+
@User() user: JwtUser,
189+
): Promise<SystemRerunResponseDto> {
190+
return await this.marathonMatchConfigService.rerunSystemTests(
191+
challengeId,
192+
user,
193+
);
194+
}
195+
151196
/**
152197
* Lists marathon match configurations with optional filters and pagination.
153198
* @param query Search and pagination query parameters.

src/api/marathon-match-config/marathon-match-config.service.spec.ts

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,9 @@ describe('MarathonMatchConfigService', () => {
5252
const prismaErrorService = {
5353
handleError: jest.fn(),
5454
};
55+
const scoringResultService = {
56+
triggerSystemScore: jest.fn(),
57+
};
5558

5659
jest.spyOn(LoggerService, 'forRoot').mockReturnValue(mockLogger as never);
5760

@@ -61,6 +64,7 @@ describe('MarathonMatchConfigService', () => {
6164
m2mService as never,
6265
prisma as never,
6366
prismaErrorService as never,
67+
scoringResultService as never,
6468
);
6569

6670
return {
@@ -70,6 +74,7 @@ describe('MarathonMatchConfigService', () => {
7074
m2mService,
7175
prisma,
7276
prismaErrorService,
77+
scoringResultService,
7378
};
7479
};
7580

@@ -597,6 +602,146 @@ describe('MarathonMatchConfigService', () => {
597602
expect(ecsService.launchScorerTask).not.toHaveBeenCalled();
598603
});
599604

605+
it('reruns all matching existing system reviews', async () => {
606+
const { service, httpService, m2mService, prisma, scoringResultService } =
607+
createService();
608+
const user = {
609+
isMachine: false,
610+
userId: '40051399',
611+
} as never;
612+
613+
prisma.marathonMatchConfig.findUnique.mockResolvedValue({
614+
id: 'config-1',
615+
challengeId: '30000123',
616+
active: true,
617+
reviewScorecardId: 'legacy-scorecard-1',
618+
testerId: 'tester-1',
619+
tester: {
620+
compilationStatus: CompilationStatus.SUCCESS,
621+
},
622+
phaseConfigs: [
623+
{
624+
id: 'phase-system',
625+
configType: PhaseConfigType.SYSTEM,
626+
phaseId: 'system-phase',
627+
startSeed: BigInt(1000),
628+
numberOfTests: 5000,
629+
},
630+
],
631+
});
632+
m2mService.getM2MToken.mockResolvedValue('m2m-token');
633+
httpService.get
634+
.mockReturnValueOnce(
635+
of({
636+
data: {
637+
status: 'ACTIVE',
638+
},
639+
}),
640+
)
641+
.mockReturnValueOnce(
642+
of({
643+
data: {
644+
id: 'canonical-scorecard-1',
645+
},
646+
}),
647+
)
648+
.mockReturnValueOnce(
649+
of({
650+
data: {
651+
result: {
652+
content: [
653+
{
654+
id: 'review-2',
655+
submissionId: 'submission-2',
656+
scorecardId: 'canonical-scorecard-1',
657+
status: 'COMPLETED',
658+
},
659+
{
660+
id: 'review-1',
661+
submissionId: 'submission-1',
662+
scoreCardId: 'legacy-scorecard-1',
663+
status: 'IN_PROGRESS',
664+
},
665+
{
666+
id: 'review-cancelled',
667+
submissionId: 'submission-cancelled',
668+
scorecardId: 'canonical-scorecard-1',
669+
status: 'CANCELLED',
670+
},
671+
{
672+
id: 'review-other',
673+
submissionId: 'submission-other',
674+
scorecardId: 'scorecard-other',
675+
status: 'PENDING',
676+
},
677+
],
678+
},
679+
},
680+
headers: {
681+
'x-total-pages': '1',
682+
},
683+
}),
684+
);
685+
scoringResultService.triggerSystemScore
686+
.mockResolvedValueOnce({
687+
taskArn: 'arn:aws:ecs:task/task-1',
688+
taskId: 'task-1',
689+
})
690+
.mockResolvedValueOnce({
691+
taskArn: 'arn:aws:ecs:task/task-2',
692+
taskId: 'task-2',
693+
});
694+
695+
const result = await service.rerunSystemTests('30000123', user);
696+
697+
expect(result).toEqual({
698+
challengeId: '30000123',
699+
reviewsQueued: 2,
700+
results: [
701+
{
702+
reviewId: 'review-1',
703+
submissionId: 'submission-1',
704+
taskArn: 'arn:aws:ecs:task/task-1',
705+
taskId: 'task-1',
706+
},
707+
{
708+
reviewId: 'review-2',
709+
submissionId: 'submission-2',
710+
taskArn: 'arn:aws:ecs:task/task-2',
711+
taskId: 'task-2',
712+
},
713+
],
714+
});
715+
expect(scoringResultService.triggerSystemScore).toHaveBeenCalledTimes(2);
716+
expect(scoringResultService.triggerSystemScore).toHaveBeenNthCalledWith(
717+
1,
718+
'review-1',
719+
'submission-1',
720+
'30000123',
721+
);
722+
expect(scoringResultService.triggerSystemScore).toHaveBeenNthCalledWith(
723+
2,
724+
'review-2',
725+
'submission-2',
726+
'30000123',
727+
);
728+
expect(httpService.get).toHaveBeenNthCalledWith(
729+
3,
730+
'https://api.topcoder-dev.com/v6/reviews',
731+
{
732+
headers: {
733+
Authorization: 'Bearer m2m-token',
734+
},
735+
params: {
736+
challengeId: '30000123',
737+
page: 1,
738+
perPage: 100,
739+
thin: 'true',
740+
},
741+
},
742+
);
743+
});
744+
600745
it('normalizes large startSeed strings to BigInt when creating phase configs', async () => {
601746
const { service, httpService, m2mService, prisma } = createService();
602747
const maxRangeStartSeed = '9223372036854775800';

0 commit comments

Comments
 (0)