test(e2e): add GPU double-onboard E2E + migrate GPU jobs to NVKS runners (NVIDIA#2606) (NVIDIA#2617)

## Summary

Add `test-gpu-double-onboard.sh` that reproduces the exact scenario from
NVIDIA#2553: re-onboard with `NEMOCLAW_PROVIDER=ollama` must not leave the
proxy running with a different token than what's persisted to disk.

### What the test covers

The exact reproduction scenario from NVIDIA#2553:
1. Full non-interactive onboard with `NEMOCLAW_PROVIDER=ollama`
2. Verify proxy running, token persisted, inference works
3. Re-onboard (second onboard, same sandbox with
`NEMOCLAW_RECREATE_SANDBOX=1`)
4. **Token consistency**: file token matches what the running proxy
accepts (not 401)
5. Inference through sandbox still works after re-onboard

### NVKS runner migration

Both `gpu-e2e` (existing) and `gpu-double-onboard-e2e` (new) are moved
from the never-enabled `self-hosted` runner to NVKS ephemeral GPU
runners (`linux-amd64-gpu-l40g-latest-1` — L40G, 48 GB VRAM):

- **Removes `GPU_E2E_ENABLED` gate** — the variable was never set, so
`gpu-e2e` has been silently skipped every nightly since it was added on
March 25
- **Eliminates `needs: gpu-e2e` serialization** — separate ephemeral
VMs, no shared state
- **Drops timeout from 60 to 30 min** — fresh VM each run, no stale
state cleanup, auto-selected model is `qwen2.5:7b` (4.4 GB pull)
- **Enables GPU E2E coverage in nightly for the first time**

### Testing

The nightly workflow only triggers on `schedule` and
`workflow_dispatch`. To validate before merge, run the workflow manually
from this branch.

Closes NVIDIA#2606

### Related
- NVIDIA#2553 — The bug (ollama proxy token divergence)
- NVIDIA#2555 — The fix (restart proxy when token diverges)
- NVIDIA#839 — Original gpu-e2e addition (never enabled)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* GPU nightly E2E workflow now runs unconditionally on ephemeral GPU
runners with a shorter timeout and improved failure reporting that
surfaces the new double-onboard job.

* **Tests**
* Added a comprehensive GPU E2E that verifies double-onboard behavior,
proxy token persistence and acceptance/rejection, end-to-end inference,
and sandbox cleanup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: jyaunches

.github/workflows/nightly-e2e.yaml

@@ -15,9 +15,8 @@
 #                            probe → live inference. Validates the multi-agent architecture.
 #   skip-permissions-e2e     Validates --dangerously-skip-permissions activates the permissive
 #                            policy (not stuck in Pending) and sandbox egress works (not 403).
-#   gpu-e2e                  Local Ollama inference on a GPU self-hosted runner.
-#                            Controlled by the GPU_E2E_ENABLED repository variable.
-#                            Set vars.GPU_E2E_ENABLED to "true" in repo settings to enable.
+#   gpu-e2e                  Local Ollama inference on an NVKS ephemeral GPU runner.
+#   gpu-double-onboard-e2e   Ollama proxy token consistency after re-onboard (#2553).
 #   notify-on-failure        Auto-creates a GitHub issue when any E2E job fails.
 #
 # Runs directly on the runner (not inside Docker) because OpenShell bootstraps
@@ -545,15 +544,12 @@ jobs:
           if-no-files-found: ignore
 
   # ── GPU E2E (Ollama local inference) ──────────────────────────
-  # Enable by setting repository variable GPU_E2E_ENABLED=true
-  # (Settings → Secrets and variables → Actions → Variables)
-  #
-  # Runner labels: using 'self-hosted' for now. Refine to
-  # [self-hosted, linux, x64, gpu] once NVIDIA runner labels are confirmed.
+  # Runs on an NVKS ephemeral GPU runner (RTX Pro 6000, 36 GB VRAM).
+  # Each job gets a fresh VM — no state leakage between runs.
   gpu-e2e:
-    if: github.repository == 'NVIDIA/NemoClaw' && vars.GPU_E2E_ENABLED == 'true'
-    runs-on: self-hosted
-    timeout-minutes: 60
+    if: github.repository == 'NVIDIA/NemoClaw'
+    runs-on: linux-amd64-gpu-rtxpro6000-latest-1
+    timeout-minutes: 30
     env:
       NEMOCLAW_NON_INTERACTIVE: "1"
       NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
@@ -594,6 +590,62 @@ jobs:
           path: /tmp/nemoclaw-gpu-e2e-test.log
           if-no-files-found: ignore
 
+  # ── GPU Double-Onboard E2E (Ollama token consistency) ────────
+  # Reproduces issue #2553: re-onboard with Ollama must not leave the
+  # proxy running with a different token than what's persisted to disk.
+  # Runs on its own ephemeral VM — no dependency on gpu-e2e.
+  gpu-double-onboard-e2e:
+    if: github.repository == 'NVIDIA/NemoClaw'
+    runs-on: linux-amd64-gpu-rtxpro6000-latest-1
+    timeout-minutes: 30
+    env:
+      NEMOCLAW_NON_INTERACTIVE: "1"
+      NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
+      NEMOCLAW_SANDBOX_NAME: "e2e-gpu-double-onboard"
+      NEMOCLAW_RECREATE_SANDBOX: "1"
+      NEMOCLAW_PROVIDER: "ollama"
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Verify GPU availability
+        run: |
+          echo "=== GPU Info ==="
+          nvidia-smi
+          echo ""
+          echo "=== VRAM ==="
+          nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
+          echo ""
+          echo "=== Docker ==="
+          docker info --format '{{.ServerVersion}}'
+
+      - name: Run GPU double-onboard E2E test
+        run: bash test/e2e/test-gpu-double-onboard.sh
+
+      - name: Upload install log on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: gpu-double-onboard-install-log
+          path: /tmp/nemoclaw-gpu-double-onboard-install.log
+          if-no-files-found: ignore
+
+      - name: Upload re-onboard log on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: gpu-double-onboard-reonboard-log
+          path: /tmp/nemoclaw-gpu-double-onboard-reonboard.log
+          if-no-files-found: ignore
+
+      - name: Upload test log on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: gpu-double-onboard-test-log
+          path: /tmp/nemoclaw-gpu-double-onboard-test.log
+          if-no-files-found: ignore
+
   notify-on-failure:
     runs-on: ubuntu-latest
     needs:
@@ -616,6 +668,7 @@ jobs:
         rebuild-hermes-e2e,
         overlayfs-autofix-e2e,
         gpu-e2e,
+        gpu-double-onboard-e2e,
       ]
     if: ${{ always() && (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')) }}
     permissions: