ci(e2e): move gateway health guard to regressions

[jyaunches]

Summary

Moves the existing #3111 gateway-health-honest-e2e coverage guard out of scheduled nightly-e2e.yaml and into the new regression-e2e.yaml holding-pen workflow.

Why

Failing-test-first guards and high-signal regression anchors should be easy to dispatch while fixes are in flight, but should not automatically keep scheduled nightly red. The new regression workflow gives us a place to keep these guards and periodically review/promote stable ones into nightly.

What changed

• Removed gateway-health-honest-e2e from nightly-e2e.yaml job list and reporting dependencies.
• Added regression-e2e.yaml with gateway-health-honest-e2e as a manually dispatchable regression job.
• Left test/e2e/test-gateway-health-honest.sh unchanged.

Validation

• YAML parse for nightly-e2e.yaml
• YAML parse for regression-e2e.yaml

Related: #3111
Related PR: #3362

Summary by CodeRabbit

• Chores
  • Removed the automated nightly E2E run for gateway-health-honest and updated nightly job selection and downstream notifications accordingly
  • Added a manual regression E2E workflow for gateway-health-honest with on-demand job selection, gating logic, concurrency control, and failure-artifact upload
  • Updated local testing guidance to recommend using the new regression workflow for gateway-health-honest runs

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9c513fd4-0a62-4769-950f-8cd97db6dce8

📥 Commits

Reviewing files that changed from the base of the PR and between b4e278e and 8ed80c5.

📒 Files selected for processing (1)

• .github/workflows/regression-e2e.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• .github/workflows/regression-e2e.yaml

---

📝 Walkthrough

Walkthrough

Removes the nightly gateway-health-honest-e2e job and adds a regression-e2e manual workflow. Updates dispatch inputs and .coderabbit.yaml; the new workflow includes a selector job, a gated gateway-health-honest-e2e job that runs the E2E script on Node.js 22, and failure-only artifact uploads.

Changes

E2E Workflow Restructuring

Layer / File(s)	Summary
Remove gateway-health-honest-e2e from nightly workflow .github/workflows/nightly-e2e.yaml	Removes gateway-health-honest-e2e from the documentation list, removes it from the workflow_dispatch jobs valid list, deletes its job block, and excludes it from notify-on-failure, report-to-pr, and scorecard needs arrays.
Create gated regression job in regression-e2e .github/workflows/regression-e2e.yaml	Adds select_regression_jobs to parse inputs.jobs and outputs booleans; gates dashboard-remote-bind-e2e and adds gateway-health-honest-e2e as a gated job restricted to NVIDIA/NemoClaw, runs checkout + Node.js 22, executes test/e2e/test-gateway-health-honest.sh, and uploads three failure-only artifacts (ignore-missing).
Update CI instruction snippet .coderabbit.yaml	Updates the E2E test recommendation for test/e2e/test-gateway-health-honest.sh to reference regression-e2e.yaml and the jobs=gateway-health-honest-e2e dispatch command.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#3410: Modifies regression E2E workflow workflow_dispatch plumbing and job gating; directly related to regression job selection changes.

Suggested reviewers

• ericksoa

Poem

🐰 A job that once ran in the night,
Now waits for a click to take flight,
Selector hums, Node starts the test,
Logs are kept when failures rest,
A hopping rabbit cheers the slight.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the primary change: moving the gateway-health-honest-e2e job from the nightly workflow to regressions. It's concise, specific, and clearly summarizes the main objective.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/move-3111-guard-to-regression-e2e

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

E2E Advisor Recommendation

Required E2E: gateway-health-honest-e2e
Optional E2E: dashboard-remote-bind-e2e

Dispatch hint: gateway-health-honest-e2e

Workflow run

Full advisor summary

Pi Semantic E2E Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• gateway-health-honest-e2e: This PR removes gateway-health-honest-e2e from nightly-e2e.yaml and re-adds it under regression-e2e.yaml with a new select_regression_jobs gating step. Running it once via the new workflow validates that (a) the job still discovers and executes test/e2e/test-gateway-health-honest.sh correctly under the relocated definition, (b) the new select_regression_jobs job-name matching includes 'gateway-health-honest-e2e' when explicitly requested, and (c) the [Linux][Install] PR #3001 Docker-driver gateway requires Ubuntu 24.04+ (GLIBC 2.39) — not documented, fails silently on Ubuntu 22.04 with false "healthy" status #3111 false-positive health-check coverage guard continues to function. Without this run we have no signal that the move did not silently disable a security/health-correctness regression guard.

Optional E2E

• dashboard-remote-bind-e2e: The dashboard-remote-bind-e2e job's gating expression was rewritten from a direct contains(format(...), inputs.jobs) check into needs.select_regression_jobs.outputs.dashboard == 'true'. A confidence run via regression-e2e.yaml with jobs=dashboard-remote-bind-e2e confirms the new shell-based job selector still routes 'dashboard-remote-bind-e2e' correctly and does not regress this previously-working job.

New E2E recommendations

• ci-workflow-self-validation (medium): There is no static test that asserts every name listed in regression-e2e.yaml's Valid: jobs input description corresponds to (1) a real job defined in the same workflow and (2) a matching branch in select_regression_jobs. As more jobs are promoted into the regression holding pen, drift between the documented input list, the select step's case branches, and the actual job ids will silently produce always-skipped jobs. A small unit-style coverage test (similar to test/validate-e2e-coverage.test.ts referenced in .coderabbit.yaml) for regression-e2e.yaml would prevent this.
  • Suggested test: test/validate-regression-e2e-coverage.test.ts — parse .github/workflows/regression-e2e.yaml, extract the comma-separated job names from the jobs input description, the case branches in select_regression_jobs, and the top-level job ids; assert all three sets are equal.

Dispatch hint

• Workflow: regression-e2e.yaml
• jobs input: gateway-health-honest-e2e

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/nightly-e2e.yaml:
- Line 70: Update the .coderabbit.yaml path_instructions for
test/e2e/test-gateway-health-honest.sh to remove the stale reference to the
removed job gateway-health-honest-e2e (or delete the whole path_instructions
entry if unused); specifically edit the path_instructions block (currently lines
~495–511) to either drop gateway-health-honest-e2e from the listed jobs or
remove that file’s path entry entirely so it no longer references the removed
job and stops causing cross-validation failures.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 997c9ea4-c58a-40c6-be64-74e2522adce5

📥 Commits

Reviewing files that changed from the base of the PR and between edb7478 and 64449a8.

📒 Files selected for processing (2)

• .github/workflows/nightly-e2e.yaml
• .github/workflows/regression-e2e.yaml

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/regression-e2e.yaml:
- Around line 77-81: The contains check is failing when inputs.jobs contains
spaces (e.g., "dashboard-remote-bind-e2e, gateway-health-honest-e2e"); normalize
inputs.jobs by stripping whitespace before matching: replace uses of
contains(format(',{0},', inputs.jobs), ',gateway-health-honest-e2e,') with
contains(format(',{0},', replace(inputs.jobs, ' ', '')),
',gateway-health-honest-e2e,') (and apply the same replace(inputs.jobs, ' ', '')
normalization to the dashboard gate check that uses inputs.jobs) so tokens match
even when users include spaces.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 74e3a5f7-0232-4d5e-92fc-bb198dfdbff2

📥 Commits

Reviewing files that changed from the base of the PR and between 39e744a and b4e278e.

📒 Files selected for processing (1)

• .github/workflows/regression-e2e.yaml

[ericksoa]

Reviewed current head 8ed80c5 after the regression-e2e job-selection fix. The whitespace-sensitive dispatch issue is fixed with an explicit selector job, YAML validates, and the PR remains CI-only.