Merge pull request #398 from topcoder-platform/PM-4299-1

PM-4299: allow rerun dispatch when ECS listing is denied

Author: jmgasper

ECS_RUNNER_LIFETIME.md

@@ -98,7 +98,7 @@ Inside the task, the trusted Java parent runner:
 1. Fetches the challenge config, tester JAR, and tester metadata from `marathon-match-api-v6`.
 2. Downloads the submission from the configured Submission API URL.
 3. Posts an initial progress callback for `PROVISIONAL` and `SYSTEM` runs.
-4. Starts the tester in an isolated child JVM. Submitted solution commands run as the separate `scorer` user with scrubbed environment, restricted filesystem access, and no outbound INET/INET6 socket creation.
+4. Starts the tester in an isolated child JVM. Submitted solution commands run as the separate `scorer` user with scrubbed environment, restricted filesystem access, and no outbound INET/INET6 socket creation. For standard generic-runner seeds, the child invokes `mm-scorer-isolate --cleanup-scorer-state` before and after each test case so scorer-owned files in fixed writable roots are reset between seeds.
 5. Uploads public/private artifacts back through Submission API.
 6. Posts the final scoring callback to `POST /v6/marathon-match/internal/scoring-results`.
 7. Exits with code `0` only after the final callback succeeds.

README.md

@@ -157,6 +157,7 @@ The ECS task still needs trusted outbound access to fetch challenge config, down
 - The trusted parent runner holds `ACCESS_TOKEN`, performs network calls, and never loads untrusted submission code directly.
 - The parent launches a separate child JVM through `mm-runner-isolate` with a scrubbed environment, so submission processes do not inherit the bearer token or other runner env vars.
 - Generic submitted solution commands run through `mm-scorer-isolate` as the separate non-root `scorer` user. Downloaded tester JARs and serialized scorer config are kept runner-owned mode `0400`, so submitted code cannot read or modify them from `/tmp`.
+- Standard generic-runner seed execution asks `mm-scorer-isolate` to reset scorer-owned writable state before and after each test case. The cleanup helper only scans fixed writable roots such as `/tmp`, `/var/tmp`, `/dev/shm`, and the scorer home, and only removes entries owned by the scorer UID.
 - Generic submitted solution commands also run under a filesystem allowlist that permits runtime/toolchain reads and scorer temp writes but does not permit reading infrastructure paths such as `/etc/hostname`, `/etc/resolv.conf`, `/proc/self/cgroup`, `/proc/self/mounts`, or proc network tables.
 - Native wrappers block `io_uring` and creation of non-`AF_UNIX` sockets for submitted solution processes and their fork/exec children, so submissions cannot open live outbound network connections.
 - Standard Topcoder Marathon testers run through the generic runner flow, which creates the callback score payload from trusted runner code. Custom tester `runTester(...)` result maps remain supported for advanced cases.

ecs-runner/README.md

@@ -19,9 +19,9 @@ This image is the runtime container for marathon match scoring tasks launched by
 - The container entrypoint starts as `root`. Do not override the ECS task-definition `user` for this container; root is needed for trusted bootstrap work and for preparing runner-owned files before the child JVM starts.
 - The trusted parent runner performs network bootstrap work: fetch challenge config, download tester/submission artifacts, upload artifacts, and post the scoring callback.
 - The tester executes in a separate child JVM launched through `mm-runner-isolate` as uid/gid `10001` (`runner`) with a scrubbed environment, so `ACCESS_TOKEN` and other runner env vars are not inherited by untrusted code.
-- Generic submitted solution commands execute through the setuid-root `mm-scorer-isolate` bridge as uid/gid `10002` (`scorer`). The bridge drops its supervisor back to the invoking `runner` uid after it forks the solution child, then supervises the solution process group so tester timeouts can still terminate lower-privilege processes.
+- Generic submitted solution commands execute through the setuid-root `mm-scorer-isolate` bridge as uid/gid `10002` (`scorer`). The bridge drops its supervisor back to the invoking `runner` uid after it forks the solution child, retaining only `CAP_KILL` so tester timeouts can still terminate the lower-privilege solution process group.
 - Downloaded tester JARs are mode `0400` runner-owned files. The serialized scorer config starts as a mode `0400` runner-owned handoff file and the child JVM deletes it immediately after loading it, before tester or submitted solution code executes.
-- Generic Marathon seed execution resets scorer-owned writable state, including `/tmp`, before and after every test case, so files written by one seed are not visible to later seeds in the same submission run.
+- Generic Marathon seed execution resets scorer-owned writable state before and after every test case, so files written by one seed are not visible to later seeds in the same submission run. The Java child invokes the setuid `mm-scorer-isolate --cleanup-scorer-state` mode, which scans only fixed writable roots such as `/tmp`, `/var/tmp`, `/dev/shm`, and the scorer home, and removes only entries owned by uid `10002`.
 - Generic submitted solution commands run under a Landlock filesystem allowlist. Runtime and toolchain files are readable, `/tmp` and the scorer home are writable, and infrastructure-revealing paths such as `/etc/hostname`, `/etc/resolv.conf`, `/proc/self/cgroup`, `/proc/self/mounts`, and proc network tables are not readable by submitted code.
 - Artifact previews and artifact zip uploads include only non-symlink regular files from the runner artifact directories. Submitted symlinks are ignored instead of being dereferenced by the trusted parent runner.
 - Submitted solution processes and their fork/exec children cannot use `io_uring` and can create only `AF_UNIX` sockets. These restrictions are kernel seccomp filters inherited across fork/exec, so clearing `LD_PRELOAD` in a spawned child process does not restore INET or INET6 socket access. Outbound network access from the submission itself is therefore blocked even though the parent runner still has the trusted egress it needs.

ecs-runner/boilerplate/src/main/java/com/topcoder/marathon/MarathonTester.java

@@ -35,6 +35,7 @@ public abstract class MarathonTester {
     private final List<BufferedWriter> solInputWriters = new ArrayList<BufferedWriter>();
     private static final int maxSolutionOutputLength = 10_000_000;
     private static final long processDrainTimeoutMillis = 250;
+    private static final long processTimeoutDestroyMillis = 250;
     private static final long errorReaderDrainTimeoutMillis = 250;
     private BufferedWriter solOutputWriter;
     private BufferedWriter solErrorWriter;
@@ -110,17 +111,20 @@ protected final void startTime() {
                 lastTimeoutThread = new Thread() {
                     public void run() {
                         try {
-                            boolean finished = process.waitFor(timeLimit - elapsedTime, TimeUnit.NANOSECONDS);
+                            long remainingTime = Math.max(1L, timeLimit - elapsedTime);
+                            boolean finished = process.waitFor(remainingTime, TimeUnit.NANOSECONDS);
                             if (!finished) {
+                                boolean notifyTimeout = false;
                                 synchronized (timeLock) {
                                     if (lastStart > 0) elapsedTime += System.nanoTime() - lastStart;
                                     lastStart = 0;
-                                    if (process != null) process.destroyForcibly();
                                     if (!timeout) {
                                         timeout = true;
-                                        timeout();
+                                        notifyTimeout = true;
                                     }
                                 }
+                                terminateTimedOutProcess();
+                                if (notifyTimeout) timeout();
                             }
                         } catch (Exception e) {
                         }
@@ -186,6 +190,73 @@ public final double runTest() {
         return score;
     }
 
+    /**
+     * Terminates the submitted solution process after its measured execution
+     * time expires and closes runner-side streams so blocked tester reads unwind.
+     *
+     * <p>The ECS scorer command is wrapped by a small native supervisor. A
+     * graceful destroy gives that wrapper a catchable signal it can relay to the
+     * low-privilege scorer process group before this method escalates to a hard
+     * process kill.
+     */
+    private void terminateTimedOutProcess() {
+        Process processToStop = process;
+        if (processToStop != null) {
+            processToStop.destroy();
+            try {
+                if (!processToStop.waitFor(processTimeoutDestroyMillis, TimeUnit.MILLISECONDS)) {
+                    processToStop.destroyForcibly();
+                    processToStop.waitFor(processDrainTimeoutMillis, TimeUnit.MILLISECONDS);
+                }
+            } catch (InterruptedException e) {
+                Thread.currentThread().interrupt();
+                processToStop.destroyForcibly();
+            } catch (Exception e) {
+                processToStop.destroyForcibly();
+            }
+        }
+
+        closeSolutionStreamsAfterTimeout();
+    }
+
+    /**
+     * Closes solution pipes after timeout termination.
+     *
+     * <p>Some failed process-tree terminations can leave descendant processes
+     * holding the stdout pipe open. Closing the Java-side reader prevents
+     * {@link #readLine()} from blocking forever after the timeout has already
+     * been recorded.
+     */
+    private void closeSolutionStreamsAfterTimeout() {
+        for (BufferedWriter out : solInputWriters) {
+            try {
+                out.close();
+            } catch (Exception e) {
+            }
+        }
+        if (solOutputReader != null) {
+            try {
+                solOutputReader.close();
+            } catch (Exception e) {
+            }
+        }
+        if (solOutputWriter != null) {
+            try {
+                solOutputWriter.close();
+            } catch (Exception e) {
+            }
+        }
+        if (solErrorReader != null) {
+            solErrorReader.closeAndWait(errorReaderDrainTimeoutMillis);
+            solutionError = solErrorReader.getOutput();
+        } else if (solErrorWriter != null) {
+            try {
+                solErrorWriter.close();
+            } catch (Exception e) {
+            }
+        }
+    }
+
     protected final void writeLine(int v) throws Exception {
         writeLine(String.valueOf(v));
     }