Add e2e test coverage spec for troubleshooting mode scenarios#2983
Add e2e test coverage spec for troubleshooting mode scenarios#2983JoaoFula wants to merge 1 commit into
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
📝 WalkthroughWalkthroughThis change adds a documentation section to agent-modes.md describing end-to-end test coverage for mode=troubleshooting, covering the cluster_scenario pytest marker, live-cluster requirements, test flow, scenario locations, and a planned lightspeed-eval quality evaluation follow-on. ChangesDocumentation Update
Estimated code review effort: 1 (Trivial) | ~2 minutes Estimated code review effort1 (Trivial) | ~2 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.ai/spec/what/agent-modes.md:
- Around line 172-180: The troubleshooting test coverage description is out of
sync with the current code layout: it references a `cluster_scenario`-gated
`tests/e2e/test_troubleshooting_scenarios.py`, while the actual troubleshooting
eval entrypoint is still the commented-out `@pytest.mark.lseval` placeholder in
`tests/e2e/evaluation/test_lseval_troubleshooting.py`. Update the spec section
to match the current implementation and use the existing
`tests/e2e/evaluation/test_lseval_troubleshooting.py`/`@pytest.mark.lseval`
symbols, or explicitly label the scenario-based coverage as planned future work
so the documentation does not point to a non-existent test entrypoint.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 30597a7e-4d3c-4701-8c94-c27f68f0acaf
📒 Files selected for processing (1)
.ai/spec/what/agent-modes.md
| Scenario-based e2e tests verify that `mode=troubleshooting` produces structurally correct responses when diagnosing real cluster problems. Tests are gated behind the `cluster_scenario` pytest marker and require `oc` access to a live OpenShift cluster. | ||
|
|
||
| Each test: | ||
| 1. Injects a broken cluster state via setup script | ||
| 2. Queries OLS with `mode=troubleshooting` | ||
| 3. Asserts HTTP 200, non-empty response, and presence of problem-domain keywords | ||
| 4. Cleans up cluster state | ||
|
|
||
| Scenarios are defined under `eval/troubleshooting/scenarios/`. The test module is `tests/e2e/test_troubleshooting_scenarios.py`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n "cluster_scenario|lseval|test_troubleshooting_scenarios|test_lseval_troubleshooting" .ai/spec/what tests/e2eRepository: openshift/lightspeed-service
Length of output: 5052
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n--- .ai/spec/what/agent-modes.md (lines 168-183) ---\n'
sed -n '168,183p' .ai/spec/what/agent-modes.md
printf '\n--- tests/e2e/evaluation/test_lseval_troubleshooting.py (lines 150-210) ---\n'
sed -n '150,210p' tests/e2e/evaluation/test_lseval_troubleshooting.py
printf '\n--- tests/e2e/pytest.ini ---\n'
sed -n '1,80p' tests/e2e/pytest.iniRepository: openshift/lightspeed-service
Length of output: 3693
Clarify whether troubleshooting coverage is current or planned.
This section points to a cluster_scenario-gated tests/e2e/test_troubleshooting_scenarios.py, but the troubleshooting eval is still a commented-out @pytest.mark.lseval placeholder in tests/e2e/evaluation/test_lseval_troubleshooting.py. Update the spec to match the current layout, or mark this as planned future work so readers do not follow a non-existent entrypoint.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.ai/spec/what/agent-modes.md around lines 172 - 180, The troubleshooting
test coverage description is out of sync with the current code layout: it
references a `cluster_scenario`-gated
`tests/e2e/test_troubleshooting_scenarios.py`, while the actual troubleshooting
eval entrypoint is still the commented-out `@pytest.mark.lseval` placeholder in
`tests/e2e/evaluation/test_lseval_troubleshooting.py`. Update the spec section
to match the current implementation and use the existing
`tests/e2e/evaluation/test_lseval_troubleshooting.py`/`@pytest.mark.lseval`
symbols, or explicitly label the scenario-based coverage as planned future work
so the documentation does not point to a non-existent test entrypoint.
|
@JoaoFula: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Description
Type of change
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit