WIP: Add post-test alerts

[albarker-rh]

What this PR does / why we need it:

Verifies 7 critical CNV alerts were not triggered during test execution. Follows the deprecated_api auto-inclusion pattern.

• KubeVirtDeprecatedAPIRequested
• LowVirtControllersCount
• LowVirtAPICount
• KubeVirtCRModified
• VirtControllerRESTErrorsHigh
• VirtHandlerRESTErrorsHigh
• HCOOperatorConditionsUnhealthy

Which issue(s) this PR fixes:

We don't check for the above critical alerts firing during test execution

Special notes for reviewer:

jira-ticket:

https://redhat.atlassian.net/browse/CNV-80353

Summary by CodeRabbit

Tests

• Added test infrastructure for post-test critical alert verification
• New CLI option to skip post-test alert validation
• Implemented alert state monitoring tests to ensure system stability after test execution

[coderabbitai]

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (1)

• WIP

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7b99f1cc-e715-4f79-8f41-a5c967d99076

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

A new tests/post_test_alerts/test_post_test_alerts.py module is added, defining a list of critical CNV alert names and a Prometheus-based test that asserts none are firing after test execution. The conftest.py is updated to register the post_test_alerts marker, add a --skip-post-test-alerts CLI option, auto-include the new directory in collection, and filter the tests appropriately.

Changes

Post-test critical alert checker

Layer / File(s)	Summary
Alert list and Prometheus test tests/post_test_alerts/test_post_test_alerts.py	Defines POST_TEST_CRITICAL_ALERTS (module-level list of alert names) and test_no_critical_alerts_after_tests, which queries Prometheus per alert name, collects any in state == "firing", and asserts the set is empty. Decorated with s390x, polarion("CNV-80353"), and order("last").
Conftest marker, CLI option, filter, and collection hook conftest.py	Adds "post_test_alerts" to every team's entry in TEAM_MARKERS; creates PostTestAlerts argument group and --skip-post-test-alerts CLI option; defines filter_post_test_alerts_tests() that removes marked tests when the flag is set or during upgrade/install runs; extends pytest_configure to auto-collect tests/post_test_alerts; calls filter_post_test_alerts_tests() in pytest_collection_modifyitems.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

new-tests

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error, 1 inconclusive)

Check name	Status	Explanation	Resolution
Stp Link Required	❌ Error	New test file tests/post_test_alerts/test_post_test_alerts.py has incomplete Jira link (CNV-80353 key only, not full URL) in module docstring. Requires Jira: https://redhat.atlassian.net/browse/CNV...	Replace line 6 with: Jira: https://redhat.atlassian.net/browse/CNV-80353 (or equivalent full Jira URL)
Title check	❓ Inconclusive	The title 'WIP: Add post-test alerts' is vague and lacks specificity about the actual change—it doesn't clarify what alerts or why they matter.	Replace with a more descriptive title under 120 chars that specifies the change, e.g., 'Add post-test verification for critical CNV alerts' or 'Add post-test alert filtering and CNV-80353 checks'.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description provides all required sections with substantive content: clear explanation of what the PR does, the problem it solves, and proper JIRA ticket reference.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Linked repositories: Your configuration references 1 linked repositories, but your current plan allows 0. Analyzed ``, skipped RedHatQE/openshift-virtualization-tests-design-docs.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-virtualization-qe-bot]

D/S test tox -e verify-tc-requirement-polarion failed: cnv-tests-tox-executor/29166

[openshift-virtualization-qe-bot-3]

Report bugs in Issues

Welcome! 🎉

This pull request will be automatically processed with the following features:

🔄 Automatic Actions

• Reviewer Assignment: Reviewers are automatically assigned based on the OWNERS file in the repository root
• Size Labeling: PR size labels (XS, S, M, L, XL, XXL) are automatically applied based on changes
• Issue Creation: A tracking issue is created for this PR and will be closed when the PR is merged or closed
• Branch Labeling: Branch-specific labels are applied to track the target branch
• Auto-verification: Auto-verified users have their PRs automatically marked as verified
• Labels: Enabled categories: branch, can-be-merged, cherry-pick, has-conflicts, hold, needs-rebase, size, verified, wip

📋 Available Commands

PR Status Management

• /wip - Mark PR as work in progress (adds WIP: prefix to title)
• /wip cancel - Remove work in progress status
• /hold - Block PR merging (approvers only)
• /hold cancel - Unblock PR merging
• /verified - Mark PR as verified
• /verified cancel - Remove verification status
• /reprocess - Trigger complete PR workflow reprocessing (useful if webhook failed or configuration changed)
• /regenerate-welcome - Regenerate this welcome message
• /security-override - Set security check runs to pass (maintainers only)
• /security-override cancel - Re-run security checks

Review & Approval

• /lgtm - Approve changes (looks good to me)
• /approve - Approve PR (approvers only)
• /assign-reviewers - Assign reviewers based on OWNERS file
• /assign-reviewer @username - Assign specific reviewer
• /check-can-merge - Check if PR meets merge requirements

Testing & Validation

• /retest tox - Run Python test suite with tox
• /retest build-container - Rebuild and test container image
• /retest verify-bugs-are-open - verify-bugs-are-open
• /retest all - Run all available tests

Container Operations

• /build-and-push-container - Build and push container image (tagged with PR number)
  • Supports additional build arguments: /build-and-push-container --build-arg KEY=value

Cherry-pick Operations

• /cherry-pick <branch> - Schedule cherry-pick to target branch when PR is merged
  • Multiple branches: /cherry-pick branch1 branch2 branch3
• /cherry-pick-retry <branch> - Retry a failed cherry-pick (merged PRs only)

Branch Management

• /rebase - Rebase this PR branch onto its base branch

Label Management

• /<label-name> - Add a label to the PR
• /<label-name> cancel - Remove a label from the PR

✅ Merge Requirements

This PR will be automatically approved when the following conditions are met:

1. Approval: /approve from at least one approver
2. LGTM Count: Minimum 2 /lgtm from reviewers
3. Status Checks: All required status checks must pass
4. No Blockers: No wip, hold, has-conflicts labels and PR must be mergeable (no conflicts)
5. Verified: PR must be marked as verified

📊 Review Process

Approvers and Reviewers

Approvers:

• dshchedr
• myakove
• rnetser
• vsibirsk

Reviewers:

• RoniKishner
• dshchedr
• rnetser
• vsibirsk

Available Labels

• hold
• verified
• wip
• lgtm
• approve

AI Features

• Cherry-Pick Conflict Resolution: Enabled (claude/claude-opus-4-6[1m])

Security Checks

• Suspicious Path Detection: Monitors paths: .claude/, .vscode/, .cursor/, .devcontainer/, .pi/, .github/workflows/, .github/actions/
• Committer Identity Check: Verifies last committer matches PR author
• Mandatory: Security checks block merge (use /security-override to bypass — maintainers only)

💡 Tips

• WIP Status: Use /wip when your PR is not ready for review
• Verification: The verified label is removed on new commits unless the push is detected as a clean rebase
• Cherry-picking: Cherry-pick labels are processed when the PR is merged
• Container Builds: Container images are automatically tagged with the PR number
• Permission Levels: Some commands require approver permissions
• Auto-verified Users: Certain users have automatic verification and merge privileges

For more information, please refer to the project documentation or contact the maintainers.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/post_test_alerts/test_post_test_alerts.py`:
- Around line 1-7: The module docstring in test_post_test_alerts.py contains
only the Jira key "CNV-80353" instead of a full Jira URL as required by coding
guidelines for traceability. Replace the line "Jira: CNV-80353" with a complete
URL in the format "Jira: https://issues.redhat.com/browse/CNV-80353" to provide
a proper link for coverage tracking and follow the requirement that RFE/Jira
links must be included as actual links in module, class, or test docstrings.
- Around line 48-49: The condition at line 48 only inspects the first element of
alerts_by_name using alerts_by_name[0]["state"], which means if a firing alert
exists in later entries of the list, it will be missed causing false negatives.
Fix this by using a loop or the any() function to check if ANY element in the
alerts_by_name list has state equal to "firing", rather than only checking the
first element at index 0. This ensures all alert samples are inspected to
determine if the alert is actively firing.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2e876e14-e533-4540-9a89-ce379cad38d7

📥 Commits

Reviewing files that changed from the base of the PR and between 2e9579f and 484db23.

📒 Files selected for processing (3)

• conftest.py
• tests/post_test_alerts/__init__.py
• tests/post_test_alerts/test_post_test_alerts.py

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[albarker-rh]

/wip cancel

[albarker-rh]

/build-and-push-container

[openshift-virtualization-qe-bot-4]

New container for quay.io/openshift-cnv/openshift-virtualization-tests:pr-5232 published

[albarker-rh]

/wip