Tweaks for issues noted during testing

Author: jmgasper

docs/full-marathon-match-test.md

@@ -76,7 +76,7 @@ The tester source can be a standard Topcoder Marathon tester with a `main(...)`
 
 For advanced create payload fields, add `challenge.createPayload` with any valid `POST /v6/challenges` body fields. The script still applies the configured Registration and Submission duration overrides.
 
-This repo includes a `scripts/fixtures/Blocks` example fixture for the TCO22 Blocks Marathon Match. Its manifest includes all sample submissions from the fixture set, including `submission-winner.zip`, plus focused runner regression submissions such as `submission-issue-287-output-limit.zip`, which should fail scoring with an output-size-limit error instead of uploading the oversized public artifact, `submission-issue-245-long-system-callback.zip`, which intentionally keeps the 5,000-test SYSTEM runner alive long enough to validate M2M token refresh before final scoring callbacks, and `submission-issue-21-filesystem-leak.zip`, which validates the submitted-solution filesystem sandbox.
+This repo includes a `scripts/fixtures/Blocks` example fixture for the TCO22 Blocks Marathon Match. Its manifest includes all sample submissions from the fixture set, including `submission-winner.zip`, plus focused runner regression submissions such as `submission-issue-287-output-limit.zip`, which should fail scoring with an output-size-limit error instead of uploading the oversized public artifact, `submission-issue-245-long-system-callback.zip`, which intentionally keeps the 5,000-test SYSTEM runner alive long enough to validate M2M token refresh before final scoring callbacks, `submission-issue-21-filesystem-leak.zip`, which validates the submitted-solution filesystem sandbox, and `submission-timeout-under-limit.zip` / `submission-timeout-over-limit.zip`, which bracket a 100 ms measured response window to confirm per-seed timeout behavior.
 
 ## Create Fixtures from Production
 

docs/marathon-match-setup.md

@@ -112,7 +112,7 @@ The POST request validates the challenge ID against challenge-api and validates
 | `relativeScoringEnabled` | boolean | Enables relative-score normalization |
 | `scoreDirection` | string | `MAXIMIZE` or `MINIMIZE` |
 | `reviewScorecardId` | string | Review API scorecard used for summations. Must resolve via review-api using either the canonical scorecard id or a legacy scorecard id. |
-| `testTimeout` | number | Per-seed tester execution timeout in milliseconds |
+| `testTimeout` | number | Per-seed measured submitted-solution execution timeout in milliseconds. Tester setup, initial input writes, artifact IO, and other runner work before the tester starts its timed section are not charged to this limit. |
 | `compileTimeout` | number | Submission compile timeout in milliseconds |
 | `systemTestTimeout` | number | Total SYSTEM scoring timeout in milliseconds. Defaults to 24 hours (`86400000`) when omitted; if the timeout fires while the ECS task is still active and the SYSTEM summation is incomplete, the API stops the task and writes a failed summation with `metadata.timed_out = true`. |
 | `taskDefinitionName` | string | ECS task definition family |

docs/marathon-processor-specification-and-scoring-terminology.md

@@ -111,7 +111,7 @@ Compile and test timeouts are configured per Marathon Match challenge.
 - `compileTimeout` controls submission compilation timeout.
 - For Java submissions, `compileTimeout` also covers class startup and static
   initializer checks performed after `javac`.
-- `testTimeout` controls per-seed tester execution timeout.
+- `testTimeout` controls the per-seed measured submitted-solution execution timeout. Runner setup, initial tester input writes, and artifact IO before the tester starts its timed section are outside this limit; a timed-out seed reports the configured limit as its runtime.
 - `systemTestTimeout` controls the total SYSTEM scoring timeout per submission. It defaults to 24 hours and causes the API to stop a still-active ECS runner and write a failed SYSTEM summation with `metadata.timed_out = true`.
 - `POST /v6/marathon-match/challenge/:challengeId/rerun/system` restarts existing non-cancelled SYSTEM reviews with the current `testTimeout`, `systemTestTimeout`, SYSTEM seed, and SYSTEM test count settings.
 
@@ -246,7 +246,7 @@ rustc --edition=2024 -O Solution.rs -o Solution
 java -jar Tester.jar -exec "env RUST_BACKTRACE=1 ./Solution" -seed 1
 ```
 
-Members may print debug information to standard error. The tester captures solution output and error text. Internal artifacts include per-seed `stdout/{seed}.txt` and `stderr/{seed}.txt` files for operator diagnostics, while member-visible EXAMPLE artifacts expose only the approved output summary for the scoring phase, including each seed and the score received for that seed.
+Members may print debug information to standard error. The tester captures solution output and error text. Internal artifacts include `compile_log.txt`, `execution-{submissionId}.log`, optional `error-{submissionId}.log`, and per-seed `stdout/{seed}.txt` and `stderr/{seed}.txt` files for operator diagnostics, while member-visible EXAMPLE artifacts expose only the approved output summary for the scoring phase, including each seed and the score received for that seed.
 
 ## Download and Access Submissions
 

docs/submission-phase-scoring.md

@@ -101,7 +101,7 @@ Runner task logs are written to CloudWatch using the submission runner log strea
 
 For `PROVISIONAL` and `SYSTEM` scoring, the runner writes progress to the phase review summation metadata through `POST /v6/marathon-match/internal/scoring-progress`. Review API exposes this as `reviewSummation.metadata.testProcess` (`provisional` or `system`), `reviewSummation.metadata.testProgress` (`0` to `1`), and `reviewSummation.metadata.testStatus` (`IN PROGRESS`, `SUCCESS`, or `FAILED`) when metadata is included in the response. In-progress summations keep a neutral placeholder score and must be rendered as unavailable based on `testStatus`; only `FAILED` progress uses the failed-score sentinel. SYSTEM timeout failures also include `reviewSummation.metadata.timed_out = true`.
 
-For generic runner artifacts, only EXAMPLE scoring uploads the member-visible public artifact. Its public `output.txt` includes each testcase ordinal, actual seed, seed score, runtime, runner/tester errors, and submitted-solution stderr, while submitted-solution stdout stays private. Every generic scoring phase uploads private internal artifacts: per-test score details remain in `reviews.json` and callback metadata, while captured submitted-solution output is split into `stdout/{seed}.txt` and `stderr/{seed}.txt` files for each actual seed. PROVISIONAL and SYSTEM scoring do not publish competitor-visible artifacts.
+For generic runner artifacts, only EXAMPLE scoring uploads the member-visible public artifact. Its public `output.txt` includes each testcase ordinal, actual seed, seed score, runtime, runner/tester errors, and submitted-solution stderr, while submitted-solution stdout stays private. Every generic scoring phase uploads private internal artifacts: per-test score details remain in `reviews.json` with testcase ordinals and actual seeds, callback metadata keeps member-visible testcase ordinals only, compile and execution logs are preserved as `compile_log.txt`, `execution-{submissionId}.log`, and optional `error-{submissionId}.log`, and captured submitted-solution output is split into `stdout/{seed}.txt` and `stderr/{seed}.txt` files for each actual seed. PROVISIONAL and SYSTEM scoring do not publish competitor-visible artifacts.
 
 When more than one phase config matches the currently open challenge phases, the handler launches one scorer task per match. This is the supported way to run both `EXAMPLE` and `PROVISIONAL` scoring from the same Submission phase.
 

ecs-runner/README.md

@@ -142,7 +142,7 @@ Once this is saved and the config is active, new scoring launches use the new EC
 
 The runner rejects output once generated public `output.txt` content or a public/private artifact archive would exceed `MM_RUNNER_MAX_OUTPUT_BYTES`. The default is `10000000` bytes. The trusted parent runner passes the resolved limit into the isolated tester child, so generated generic-runner output and uploaded artifact zips use the same cap.
 
-Generic runner member-visible artifacts are uploaded only for EXAMPLE scoring. Their public `output.txt` entries show testcase ordinal, actual seed, seed score, runtime, runner/tester errors, and submitted-solution stderr, while intentionally omitting submitted-solution stdout. Every generic scoring phase uploads private internal artifacts in the `*-internal` zip: `reviews.json` carries per-test `testScores`, and `stdout/{seed}.txt` plus `stderr/{seed}.txt` carry captured submitted-solution output for each actual seed. PROVISIONAL and SYSTEM scoring do not publish competitor-visible artifacts.
+Generic runner member-visible artifacts are uploaded only for EXAMPLE scoring. Their public `output.txt` entries show testcase ordinal, actual seed, seed score, runtime, runner/tester errors, and submitted-solution stderr, while intentionally omitting submitted-solution stdout. Every generic scoring phase uploads private internal artifacts in the `*-internal` zip: `reviews.json` carries per-test `metadata.testScores` and top-level `testScores` with testcase ordinals and actual seeds, `compile_log.txt` and `execution-{submissionId}.log` preserve runner diagnostics, `error-{submissionId}.log` is included when present, and `stdout/{seed}.txt` plus `stderr/{seed}.txt` carry captured submitted-solution output for each actual seed. PROVISIONAL and SYSTEM scoring do not publish competitor-visible artifacts.
 
 ## Local smoke test
 

ecs-runner/boilerplate/src/main/java/com/topcoder/marathon/MarathonTester.java

@@ -101,41 +101,37 @@ public final boolean isTimeout() {
         return timeout;
     }
 
+    /**
+     * Starts measuring submitted-solution execution time.
+     *
+     * <p>Process startup, tester setup, and initial input writes that happen
+     * before this call are intentionally outside the configured timeout. The
+     * timeout watcher uses the same monotonic start instant as the elapsed-time
+     * counter so it cannot consume part of the submitted solution's configured
+     * runtime before measurement begins.
+     */
     protected final void startTime() {
         synchronized (timeLock) {
             if (lastStart != 0) {
                 System.out.println("ERROR startTime() was called again, before endTime() closed the first one.");
                 System.exit(-1);
             }
+            lastStart = System.nanoTime();
             if (timeLimit > 0) {
+                final long timeoutDeadline = lastStart + Math.max(1L, timeLimit - elapsedTime);
                 lastTimeoutThread = new Thread() {
                     public void run() {
-                        try {
-                            long remainingTime = Math.max(1L, timeLimit - elapsedTime);
-                            boolean finished = process.waitFor(remainingTime, TimeUnit.NANOSECONDS);
-                            if (!finished) {
-                                boolean notifyTimeout = false;
-                                synchronized (timeLock) {
-                                    if (lastStart > 0) elapsedTime += System.nanoTime() - lastStart;
-                                    lastStart = 0;
-                                    if (!timeout) {
-                                        timeout = true;
-                                        notifyTimeout = true;
-                                    }
-                                }
-                                terminateTimedOutProcess();
-                                if (notifyTimeout) timeout();
-                            }
-                        } catch (Exception e) {
-                        }
+                        waitForTimeoutDeadline(timeoutDeadline);
                     }
                 };
                 lastTimeoutThread.start();
             }
-            lastStart = System.nanoTime();
         }
     }
 
+    /**
+     * Stops measuring submitted-solution execution time.
+     */
     protected final void stopTime() {
         synchronized (timeLock) {
             if (lastStart > 0) elapsedTime += System.nanoTime() - lastStart;
@@ -158,12 +154,7 @@ public final double runTest() {
             score = getErrorScore();
             score = run();
             if (timeLimit > 0 && elapsedTime > timeLimit) {
-                synchronized (timeLock) {
-                    if (!timeout) {
-                        timeout = true;
-                        timeout();
-                    }
-                }
+                if (recordTimeout(false)) timeout();
             }
             end();
         } catch (Exception e) {
@@ -190,6 +181,51 @@ public final double runTest() {
         return score;
     }
 
+    /**
+     * Waits for the submitted process to exit or for the measured timeout
+     * deadline to pass.
+     *
+     * @param timeoutDeadline Monotonic {@link System#nanoTime()} deadline for
+     *                        the current measured execution interval.
+     */
+    private void waitForTimeoutDeadline(long timeoutDeadline) {
+        Process watchedProcess = process;
+        if (watchedProcess == null) return;
+
+        try {
+            while (true) {
+                long remainingTime = timeoutDeadline - System.nanoTime();
+                if (remainingTime <= 0) break;
+                if (watchedProcess.waitFor(remainingTime, TimeUnit.NANOSECONDS)) return;
+            }
+
+            if (recordTimeout(true)) {
+                terminateTimedOutProcess();
+                timeout();
+            }
+        } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+        } catch (Exception e) {
+        }
+    }
+
+    /**
+     * Records that the configured measured execution timeout was reached.
+     *
+     * @param requireActiveInterval True when timeout should only be recorded if
+     *                              {@link #startTime()} is still active.
+     * @return True when this call transitioned the tester into timeout state.
+     */
+    private boolean recordTimeout(boolean requireActiveInterval) {
+        synchronized (timeLock) {
+            if (timeout || (requireActiveInterval && lastStart == 0)) return false;
+            if (timeLimit > 0) elapsedTime = timeLimit;
+            lastStart = 0;
+            timeout = true;
+            return true;
+        }
+    }
+
     /**
      * Terminates the submitted solution process after its measured execution
      * time expires and closes runner-side streams so blocked tester reads unwind.