Skip to content

Commit adf91da

Browse files
jyaunchescv
andauthored
test(e2e): migrate test-diagnostics.sh to vitest (#5347)
## Summary Migrate `test/e2e/test-diagnostics.sh` with the simplest equivalent live Vitest coverage. Legacy shell deletion and nightly shell lane retirement are deferred to #5098 Phase 11; this PR adds the replacement Vitest path beside the stable legacy lane. ## Related Issues Refs #5098 ## Contract mapping - Legacy assertion: `nemoclaw --version` exits 0 and prints semver. - Replacement: `test/e2e-scenario/live/diagnostics.test.ts` runs the repo CLI entrypoint and asserts semver output. - Boundary preserved: real built CLI process. - Legacy assertion: `nemoclaw debug --quick --output` creates a non-empty archive within the quick timeout. - Replacement: live Vitest invokes `debug --quick --output` and checks archive existence/size/time. - Boundary preserved: real debug command, host filesystem, tarball artifact path. - Legacy assertion: full `nemoclaw debug --output` creates an extractable tarball with no NVIDIA credential leaks. - Replacement: live Vitest creates/extracts the archive and scans extracted files for the exact key and `nvapi-*` patterns. - Boundary preserved: real debug command, `tar`, archive extraction, credential scanning. - Legacy assertion: registered `--sandbox` succeeds while unknown `--sandbox` fails, names the sandbox, reports `not registered`, and writes no partial tarball. - Replacement: live Vitest onboards a real sandbox, runs both debug paths, and checks the error/archive contract. - Boundary preserved: Docker/OpenShell sandbox registry and debug sandbox validation. - Legacy assertion: sandbox `openclaw.json` is readable and host status shows model/provider diagnostics. - Replacement: live Vitest uses real OpenShell sandbox exec plus `nemoclaw <sandbox> status`. - Boundary preserved: sandbox exec and host CLI status. - Legacy assertion: `credentials list` hides values and `credentials reset` removes the provider credential. - Replacement: live Vitest captures raw process output for leak assertions, then records redacted fixture artifacts and resets `nvidia-prod` when present. - Boundary preserved: OpenShell gateway-backed credential list/reset. ## Simplicity check - Test shape: simple live Vitest test. - Original runner/lane: `nightly-e2e.yaml` job `diagnostics-e2e` via `e2e-script.yaml`, default `ubuntu-latest`, Docker/OpenShell, `NVIDIA_API_KEY`, `NEMOCLAW_RECREATE_SANDBOX=1`, 45-minute reusable timeout. - Replacement runner: same `ubuntu-latest` runner class with Docker/OpenShell and `NVIDIA_API_KEY` in `e2e-vitest-scenarios.yaml` job `diagnostics-vitest`. - New shared helpers: none; one-off helpers stay local to `diagnostics.test.ts`. - New framework/registry/ledger: **none**. - Workflow changes: add selective free-standing Vitest job and inventory mapping; preserve legacy shell wiring for Phase 11. - Selective dispatch: `e2e-vitest-scenarios.yaml` with `jobs=diagnostics-vitest` or `scenarios=diagnostics`. ## Verification - `npm ci --ignore-scripts` - `npm run build:cli` - `npm run typecheck:cli` - `NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run test/e2e-scenario/live/diagnostics.test.ts --project e2e-scenarios-live` (local no-secret run skips with evidence) - `npx vitest run test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts --project e2e-vitest-support` - `git diff --check` - Branch refreshed onto current `origin/main`; support workflow test re-ran locally after refresh. - PR: #5347 - Same-runner selective run: https://github.com/NVIDIA/NemoClaw/actions/runs/27594970645 (`workflow_dispatch`, `jobs=diagnostics-vitest`) — passed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added an end-to-end live diagnostics scenario covering sandbox behavior, debug archive creation, and verification that secrets and API-credential patterns remain fully redacted. * Validates diagnostics archive acceptance/rejection and credential listing/reset flows. * **Chores** * Extended CI to run the new diagnostics job and include its results in PR reporting, with artifact collection for review. * Updated workflow boundary validation to enforce stricter security and artifact-upload policies for the diagnostics job. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
1 parent 71bd969 commit adf91da

4 files changed

Lines changed: 680 additions & 3 deletions

File tree

.github/workflows/e2e-vitest-scenarios.yaml

Lines changed: 55 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2835,6 +2835,59 @@ jobs:
28352835
docker logout docker.io || true
28362836
rm -rf "${DOCKER_CONFIG}"
28372837
2838+
diagnostics-vitest:
2839+
needs: generate-matrix
2840+
if: ${{ (inputs.jobs == '' && inputs.scenarios == '') || contains(format(',{0},', inputs.jobs), ',diagnostics-vitest,') || contains(format(',{0},', inputs.scenarios), ',diagnostics,') }}
2841+
runs-on: ubuntu-latest
2842+
timeout-minutes: 60
2843+
env:
2844+
FREE_STANDING_VITEST_JOB: "1"
2845+
FREE_STANDING_SCENARIO_ID: "diagnostics"
2846+
E2E_ARTIFACT_DIR: ${{ github.workspace }}/e2e-artifacts/vitest/diagnostics
2847+
NEMOCLAW_CLI_BIN: ${{ github.workspace }}/bin/nemoclaw.js
2848+
NEMOCLAW_RUN_E2E_SCENARIOS: "1"
2849+
NEMOCLAW_NON_INTERACTIVE: "1"
2850+
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
2851+
NEMOCLAW_SANDBOX_NAME: "e2e-diag"
2852+
OPENSHELL_GATEWAY: "nemoclaw"
2853+
steps:
2854+
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
2855+
with:
2856+
persist-credentials: false
2857+
2858+
- name: Set up Node
2859+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.0.0
2860+
with:
2861+
node-version: 22
2862+
cache: npm
2863+
2864+
- name: Install root dependencies
2865+
run: npm ci --ignore-scripts
2866+
2867+
- name: Build CLI
2868+
run: npm run build:cli
2869+
2870+
- name: Run diagnostics live test
2871+
# Migrated from test/e2e/test-diagnostics.sh. This preserves the
2872+
# ubuntu-latest + Docker/OpenShell + NVIDIA_API_KEY lane by running
2873+
# debug archives, install.sh/onboard, sandbox exec/status, and
2874+
# gateway-backed credentials list/reset from Vitest.
2875+
env:
2876+
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
2877+
run: |
2878+
set -euo pipefail
2879+
npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/diagnostics.test.ts --silent=false --reporter=default
2880+
2881+
- name: Upload diagnostics artifacts
2882+
if: always()
2883+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
2884+
with:
2885+
name: e2e-vitest-scenarios-diagnostics
2886+
path: e2e-artifacts/vitest/diagnostics/
2887+
include-hidden-files: false
2888+
if-no-files-found: ignore
2889+
retention-days: 14
2890+
28382891
gateway-drift-preflight-vitest:
28392892
needs: generate-matrix
28402893
if: ${{ (inputs.jobs == '' && inputs.scenarios == '') || contains(format(',{0},', inputs.jobs), ',gateway-drift-preflight-vitest,') || contains(format(',{0},', inputs.scenarios), ',gateway-drift-preflight,') }}
@@ -2871,9 +2924,7 @@ jobs:
28712924
# Docker/OpenShell mutation.
28722925
run: |
28732926
set -euo pipefail
2874-
npx vitest run --project cli \
2875-
test/gateway-drift-preflight.test.ts \
2876-
--silent=false --reporter=default
2927+
npx vitest run --project cli test/gateway-drift-preflight.test.ts --silent=false --reporter=default
28772928
28782929
- name: Upload gateway drift preflight artifacts
28792930
if: always()
@@ -3561,6 +3612,7 @@ jobs:
35613612
onboard-resume-vitest,
35623613
model-router-provider-routed-inference-vitest,
35633614
sandbox-survival-vitest,
3615+
diagnostics-vitest,
35643616
snapshot-commands-vitest,
35653617
gateway-drift-preflight-vitest,
35663618
openclaw-tui-chat-correlation-vitest,

0 commit comments

Comments
 (0)