Commit adf91da
test(e2e): migrate test-diagnostics.sh to vitest (#5347)
## Summary
Migrate `test/e2e/test-diagnostics.sh` with the simplest equivalent live
Vitest coverage.
Legacy shell deletion and nightly shell lane retirement are deferred to
#5098 Phase 11; this PR adds the replacement Vitest path beside the
stable legacy lane.
## Related Issues
Refs #5098
## Contract mapping
- Legacy assertion: `nemoclaw --version` exits 0 and prints semver.
- Replacement: `test/e2e-scenario/live/diagnostics.test.ts` runs the
repo CLI entrypoint and asserts semver output.
- Boundary preserved: real built CLI process.
- Legacy assertion: `nemoclaw debug --quick --output` creates a
non-empty archive within the quick timeout.
- Replacement: live Vitest invokes `debug --quick --output` and checks
archive existence/size/time.
- Boundary preserved: real debug command, host filesystem, tarball
artifact path.
- Legacy assertion: full `nemoclaw debug --output` creates an
extractable tarball with no NVIDIA credential leaks.
- Replacement: live Vitest creates/extracts the archive and scans
extracted files for the exact key and `nvapi-*` patterns.
- Boundary preserved: real debug command, `tar`, archive extraction,
credential scanning.
- Legacy assertion: registered `--sandbox` succeeds while unknown
`--sandbox` fails, names the sandbox, reports `not registered`, and
writes no partial tarball.
- Replacement: live Vitest onboards a real sandbox, runs both debug
paths, and checks the error/archive contract.
- Boundary preserved: Docker/OpenShell sandbox registry and debug
sandbox validation.
- Legacy assertion: sandbox `openclaw.json` is readable and host status
shows model/provider diagnostics.
- Replacement: live Vitest uses real OpenShell sandbox exec plus
`nemoclaw <sandbox> status`.
- Boundary preserved: sandbox exec and host CLI status.
- Legacy assertion: `credentials list` hides values and `credentials
reset` removes the provider credential.
- Replacement: live Vitest captures raw process output for leak
assertions, then records redacted fixture artifacts and resets
`nvidia-prod` when present.
- Boundary preserved: OpenShell gateway-backed credential list/reset.
## Simplicity check
- Test shape: simple live Vitest test.
- Original runner/lane: `nightly-e2e.yaml` job `diagnostics-e2e` via
`e2e-script.yaml`, default `ubuntu-latest`, Docker/OpenShell,
`NVIDIA_API_KEY`, `NEMOCLAW_RECREATE_SANDBOX=1`, 45-minute reusable
timeout.
- Replacement runner: same `ubuntu-latest` runner class with
Docker/OpenShell and `NVIDIA_API_KEY` in `e2e-vitest-scenarios.yaml` job
`diagnostics-vitest`.
- New shared helpers: none; one-off helpers stay local to
`diagnostics.test.ts`.
- New framework/registry/ledger: **none**.
- Workflow changes: add selective free-standing Vitest job and inventory
mapping; preserve legacy shell wiring for Phase 11.
- Selective dispatch: `e2e-vitest-scenarios.yaml` with
`jobs=diagnostics-vitest` or `scenarios=diagnostics`.
## Verification
- `npm ci --ignore-scripts`
- `npm run build:cli`
- `npm run typecheck:cli`
- `NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run
test/e2e-scenario/live/diagnostics.test.ts --project e2e-scenarios-live`
(local no-secret run skips with evidence)
- `npx vitest run
test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts --project
e2e-vitest-support`
- `git diff --check`
- Branch refreshed onto current `origin/main`; support workflow test
re-ran locally after refresh.
- PR: #5347
- Same-runner selective run:
https://github.com/NVIDIA/NemoClaw/actions/runs/27594970645
(`workflow_dispatch`, `jobs=diagnostics-vitest`) — passed
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added an end-to-end live diagnostics scenario covering sandbox
behavior, debug archive creation, and verification that secrets and
API-credential patterns remain fully redacted.
* Validates diagnostics archive acceptance/rejection and credential
listing/reset flows.
* **Chores**
* Extended CI to run the new diagnostics job and include its results in
PR reporting, with artifact collection for review.
* Updated workflow boundary validation to enforce stricter security and
artifact-upload policies for the diagnostics job.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>1 parent 71bd969 commit adf91da
4 files changed
Lines changed: 680 additions & 3 deletions
File tree
- .github/workflows
- test/e2e-scenario
- live
- support-tests
- tools/e2e-scenarios
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2835 | 2835 | | |
2836 | 2836 | | |
2837 | 2837 | | |
| 2838 | + | |
| 2839 | + | |
| 2840 | + | |
| 2841 | + | |
| 2842 | + | |
| 2843 | + | |
| 2844 | + | |
| 2845 | + | |
| 2846 | + | |
| 2847 | + | |
| 2848 | + | |
| 2849 | + | |
| 2850 | + | |
| 2851 | + | |
| 2852 | + | |
| 2853 | + | |
| 2854 | + | |
| 2855 | + | |
| 2856 | + | |
| 2857 | + | |
| 2858 | + | |
| 2859 | + | |
| 2860 | + | |
| 2861 | + | |
| 2862 | + | |
| 2863 | + | |
| 2864 | + | |
| 2865 | + | |
| 2866 | + | |
| 2867 | + | |
| 2868 | + | |
| 2869 | + | |
| 2870 | + | |
| 2871 | + | |
| 2872 | + | |
| 2873 | + | |
| 2874 | + | |
| 2875 | + | |
| 2876 | + | |
| 2877 | + | |
| 2878 | + | |
| 2879 | + | |
| 2880 | + | |
| 2881 | + | |
| 2882 | + | |
| 2883 | + | |
| 2884 | + | |
| 2885 | + | |
| 2886 | + | |
| 2887 | + | |
| 2888 | + | |
| 2889 | + | |
| 2890 | + | |
2838 | 2891 | | |
2839 | 2892 | | |
2840 | 2893 | | |
| |||
2871 | 2924 | | |
2872 | 2925 | | |
2873 | 2926 | | |
2874 | | - | |
2875 | | - | |
2876 | | - | |
| 2927 | + | |
2877 | 2928 | | |
2878 | 2929 | | |
2879 | 2930 | | |
| |||
3561 | 3612 | | |
3562 | 3613 | | |
3563 | 3614 | | |
| 3615 | + | |
3564 | 3616 | | |
3565 | 3617 | | |
3566 | 3618 | | |
| |||
0 commit comments