Add CI Monitor: context-aware CI/Prow monitoring with adaptive polling and failure analysis by PillaiManish · Pull Request #60 · openshift-eng/oape-ai-e2e

PillaiManish · 2026-05-18T08:49:29Z

Summary

Adds a new /oape:ci-monitor skill that monitors CI/Prow job status for OpenShift operator PRs with adaptive polling, context-aware failure classification, and optional fix-push-rewatch loop
Adds a new UI page (/ci-monitor) and API endpoint (POST /api/v1/ci-monitor) for triggering CI monitoring from the web interface
Refactors the original monolithic ci-monitor command into a command + skill split, consistent with other commands in the repo (analyze-rfe, predict-regressions)

What changed

Skill + Command refactor (plugins/oape/):

commands/ci-monitor.md: Slimmed to prechecks + delegation (was 496 lines, now 272)
skills/ci-monitor/SKILL.md: New skill with polling algorithm, evidence collection, failure classification, root cause tracing, report template, and fix-push-rewatch protocol

Key improvements over original ci-monitor:

Release repo context: fetches ci-operator config from openshift/release for authoritative job classification (required/optional, fast/slow, cloud provider, OCP release version)
Operator repo context: reads go.mod and framework detection (local clone or GitHub API fallback)
PR change context: fetches changed file list for error-to-file correlation
On-demand diff fetch for failing files
Root cause tracing protocol: traces each failure to its origin (PR code, repo, release config, Vault, CI infra)
Auto-retest for infra flakes (max 2 per session, with guardrails)
Progress reporting during long polling sessions
Parallel PR polling with early completion
Expanded failure mode detection (install, test, build, lint, infra)
Error signature hashing for deterministic fix-loop termination
Branch verification before fix push

UI + Server (go-server/, agent/):

New UI page: go-server/static/ci-monitor.html (accessible at /ci-monitor)
New handler: go-server/ci_monitor_handler.go
New K8s job creator: go-server/ci_monitor_k8s.go
New agent module: agent/ci_monitor.py (autonomous execution, no user prompts)
Minimal wiring in existing files: 2 routes in main.go, embed directive in handlers.go, workflow type branching in main.py

Test plan

Verify /ci-monitor page loads and form validates PR URLs (1-3 inputs)
Verify POST /api/v1/ci-monitor rejects invalid PR URLs and empty input
Verify existing / page and POST /api/v1/workflows are unaffected
Run /oape:ci-monitor <pr-url> via Claude Code CLI and verify report includes root cause traces
Deploy to K8s cluster and verify end-to-end: UI → K8s Job → agent → ci-monitor skill → report

Reference

#51 - this PR is created on top of this changes

…ected failures

- Extract ci-monitor skill from monolithic command - Add release repo context (ci-operator config + step registry) - Add operator repo context and PR change context - Add root cause tracing protocol - Add auto-retest, progress reporting, parallel polling Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

openshift-ci · 2026-05-18T08:49:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: PillaiManish
Once this PR has been reviewed and has the lgtm label, please assign rausingh-rh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

PillaiManish · 2026-05-18T08:50:51Z

/ok-to-test

PillaiManish · 2026-05-18T08:51:39Z

/cc @shivprakashmuley @rausingh-rh @siddhibhor-56 improved the ci-monitor + added the UI for easy usage. PTAL.

The ci-monitor entrypoint (main.py) imports ci_monitor.py but it was missing from the COPY directive, causing ModuleNotFoundError at runtime when WORKFLOW_TYPE=ci-monitor. Co-authored-by: Cursor <cursoragent@cursor.com>

- Add jq to Dockerfile for CI use (avoids runtime download) - Add --ignore-context flag to skip self-monitoring in CI - Add --post-comment flag to post report as GitHub PR comment - Set --max-fix-rounds 0 for CI use (report-only, no push) Co-authored-by: Cursor <cursoragent@cursor.com>

swghosh

Made a fair pass over the structure and the overall design bits,

would want to review once more in detail for the actual Claude Skill and Commands to see what prompts are being used.

Overall looks in shape, as long as we can have a dedicated workflow for this. Let's also keep in mind to make sure this is re-usable for other PRs where some tests are failing and wasn't authored by our operator-dev workflow.

swghosh · 2026-06-02T09:16:47Z

we should omit this file.

swghosh · 2026-06-02T10:25:28Z

 6. Run `/oape:review OCPBUGS-0 {repo_info['base_branch']}` to review and auto-fix issues
 7. Commit all changes with a descriptive message
 8. Push the branch and create a PR against `{repo_info['base_branch']}`
+9. Run `/oape:ci-monitor <pr1-url> --timeout-min 120 --max-fix-rounds 2`


I don't think it's a feasible idea to run directly in the workflow, because it would cause the workflow to wait for a very long period of time blocking the flow.

Could be trigger this as a separate workflow instead, please?

swghosh · 2026-06-02T10:27:28Z

-    if not ep_url or not repo or not base_branch:
-        print("ERROR: EP_URL, REPO, BASE_BRANCH environment variables are required", file=sys.stderr)
-        sys.exit(1)
+    if workflow_type == "ci-monitor":


yup, I believe the intent was to implement something similar (i.e. run as a separate workflow than the primary one) but if possible we can de-duplicate the parts into 2 separate .py files. The common code can be moved to agent.py or the alike.

swghosh · 2026-06-02T10:30:29Z

while it's not a bad idea to have an UI, because this actually a monitor monitor - having a dashboard makes total sense.

The only concern is that we need infra maintenance mandatory in that case. Maybe we could move this elsewhere?

(separate change ofc, but we can get rid of the existing UI bits in go-server/static entirely too; motivation: Slack thread)

Perhaps, 👀 into https://gitlab.cee.redhat.com/service/app-interface#visual-app-interface for a possible UI integration. This could be helpful for hosting a dashboard or UI for long term.

swghosh · 2026-06-02T10:55:40Z

+        ),
+        cwd=working_dir,
+        permission_mode="bypassPermissions",
+        allowed_tools=CONFIGS["claude_allowed_tools"],


Recently, Claude code CLI has the ability to trigger a cron directly in the Claude session.

We should verify in Claude Agent SDK really has that capability too - if it does we can reuse it where Claude writes the script to watch the CI jobs every few quantum duration, and only invoke when actually failing..

https://code.claude.com/docs/en/scheduled-tasks#let-claude-choose-the-interval

[more deterministic behaviour] If not, we can write a script ourselves keep it here and only let Claude kick-off a session when jobs are actually failing.

shivprakashmuley and others added 3 commits May 1, 2026 16:29

MG-661: Added ci-monitor to monitor ci-jobs and suggest fixes for det…

b728e1f

…ected failures

Add CI monitor UI endpoint and agent entrypoint

a2f277e

Co-authored-by: Cursor <cursoragent@cursor.com>

openshift-ci Bot requested review from mytreya-rh and neha037 May 18, 2026 08:49

openshift-ci Bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 18, 2026

PillaiManish and others added 2 commits May 19, 2026 12:36

fix(Dockerfile): include ci_monitor.py in agent-worker image

3fdff9f

The ci-monitor entrypoint (main.py) imports ci_monitor.py but it was missing from the COPY directive, causing ModuleNotFoundError at runtime when WORKFLOW_TYPE=ci-monitor. Co-authored-by: Cursor <cursoragent@cursor.com>

swghosh reviewed Jun 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add CI Monitor: context-aware CI/Prow monitoring with adaptive polling and failure analysis#60

Add CI Monitor: context-aware CI/Prow monitoring with adaptive polling and failure analysis#60
PillaiManish wants to merge 5 commits into
openshift-eng:mainfrom
PillaiManish:ci-monitor

PillaiManish commented May 18, 2026 •

edited

Loading

Uh oh!

openshift-ci Bot commented May 18, 2026

Uh oh!

PillaiManish commented May 18, 2026

Uh oh!

PillaiManish commented May 18, 2026

Uh oh!

swghosh left a comment

Uh oh!

swghosh Jun 2, 2026

Uh oh!

swghosh Jun 2, 2026

Uh oh!

swghosh Jun 2, 2026

Uh oh!

swghosh Jun 2, 2026

Uh oh!

swghosh Jun 15, 2026

Uh oh!

swghosh Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

PillaiManish commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Test plan

Reference

Uh oh!

openshift-ci Bot commented May 18, 2026

Uh oh!

PillaiManish commented May 18, 2026

Uh oh!

PillaiManish commented May 18, 2026

Uh oh!

swghosh left a comment

Choose a reason for hiding this comment

Uh oh!

swghosh Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

swghosh Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

swghosh Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

swghosh Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

swghosh Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

swghosh Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PillaiManish commented May 18, 2026 •

edited

Loading