Skip to content

Commit 8c7a147

Browse files
committed
Add a script to fetch periodic failures and update the skill to use it
1 parent ec7fbb2 commit 8c7a147

2 files changed

Lines changed: 268 additions & 31 deletions

File tree

Lines changed: 40 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,62 @@
11
---
22
name: rota-check-periodic-jobs
3-
description: Analyze recent GraalPy periodic job failure Jira tickets for ROTA. Use when asked to triage, summarize, or plan work for recent periodic-job-failures issues, including date-bounded Jira searches with gdev-cli, issue detail inspection, hypotheses, reproduction commands, and implementation order.
3+
description: Analyze current GraalPy periodic job failures for ROTA. Use when asked to triage, summarize, or plan work for current periodic job failures, starting from scripts/rota_ci_failures.py output, validating linked Jira issues, inspecting logs, forming hypotheses, reproduction commands, and implementation order.
44
---
55

66
# ROTA Periodic Job Check
77

88
## Overview
9-
Triage recent GraalPy periodic job failure Jira tickets and produce implementation-ready plans.
9+
Triage current GraalPy periodic job failures and produce implementation-ready plans.
1010

1111
## Workflow
12-
1. Verify creator identity mapping:
13-
- Treat `ol-automation_ww` as Jira username `olauto`.
14-
- If a query returns zero results unexpectedly, test both identities, then keep `creator = olauto` once verified.
15-
16-
2. Filter to recent periodic job failures, excluding in-progress or closed issues:
17-
- Default to the last 14 days unless the user specifies otherwise.
18-
- Always state concrete start and end calendar dates in the response.
12+
1. Verify dashboard environment and run the periodic failure collector:
13+
- This workflow starts from `scripts/rota_ci_failures.py`, not from a Jira search.
14+
- The script requires `OTDASHBOARD_URL` and `OTDASHBOARD_TOKEN`.
15+
- If either variable is missing, stop and ask the user to set the missing variable(s). Do not fall back to querying Jira for the failure list.
16+
- Run from the repository root:
1917
```bash
20-
gdev-cli jira search --json --max 100 \
21-
-f key,summary,creator,created,status,labels,components,assignee \
22-
-jql "project = GR AND component = Python AND creator = olauto AND labels = periodic-job-failures AND created >= -14d AND status != Closed AND status != 'In Progress' ORDER BY created DESC"
18+
scripts/rota_ci_failures.py
2319
```
2420

25-
3. Fetch shortlisted issue details with `get-issue`:
21+
2. Parse the script output:
22+
- If it reports no failed jobs, report that there are no current failed periodic jobs and stop.
23+
- For each failed row, capture target, job name, last successful run, Jira ID(s), and log URL.
24+
- If a failed row has no Jira ID, flag it in the report and continue log analysis.
25+
26+
3. Validate every reported Jira issue:
27+
- Fetch each Jira issue linked by the script output:
28+
```bash
29+
gdev-cli jira get-issue --json -id GR-XXXX
30+
```
31+
- Check that the Jira matches the current failure:
32+
- The issue summary or description should identify the same error signature/root cause from the current log.
33+
- Check that the Jira is not too broad:
34+
- A generic timeout/build-failure ticket is acceptable only if it names this job or an intentionally scoped equivalent set of jobs.
35+
- A ticket covering unrelated jobs, unrelated targets, or unrelated error signatures is too broad.
36+
- Check that the Jira has component `Python`.
37+
- Notify the user about every Jira that fails any of these checks. Include the Jira key and the failed check(s).
38+
39+
4. Inspect failed job logs:
40+
- Use the `log URL` from the script output. For Buildbot URLs, fetch the executor log with:
2641
```bash
27-
gdev-cli jira get-issue --json -id GR-XXXX \
28-
| jq '{key, summary:.fields.summary, status:.fields.status.name, created:.fields.created, labels:.fields.labels, assignee:(.fields.assignee.name // null), description:.fields.description, comments:(.fields.comment.comments | map({author:.author.name, created, body}))}'
42+
gdev-cli buildbot get-log BUILD_ID
2943
```
44+
- Use `gdev-cli buildbot rca --build BUILD_ID --wait` when useful, but still inspect the relevant raw log lines.
45+
- Identify the exact failing command, error signature, first meaningful failure, and whether later errors are cleanup fallout.
3046

31-
4. Convert findings into an implementation-ready plan per issue:
47+
5. Convert findings into an implementation-ready plan per failure:
3248
- Extract failing job name, error signature, and log clue.
3349
- Map probable source area in repo.
3450
- Propose the first verification command.
35-
- Define exit criteria to close the ticket.
36-
- Prepare a temporary git worktree per issue with branch naming based on Jira key plus a very short hyphenated description.
51+
- Define exit criteria to close or update the linked ticket.
3752

3853
## Output Contract
39-
Return exactly:
40-
1. Query scope used: component, creator, time window, status filter.
41-
2. Count summary: total recent automation issues vs periodic failures.
42-
3. Issue list with key, created date, summary, and status.
43-
4. Per-issue plan with:
44-
- Hypothesis
45-
- First code locations to inspect
46-
- First reproducibility command
47-
- Exit criteria for closing ticket
48-
5. Recommended implementation order.
54+
Group the output by the Jira issue. For each, report:
55+
- The issue summary, status and assignee
56+
- The failed jobs in a table with job name, last successful run and log URL.
57+
- The analysis of the failure and the proposed plan
4958

5059
## Guardrails
51-
- State concrete dates for recency windows.
52-
- Prefer `--json` and explicit `-f` fields in searches.
53-
- Use `get-issue` only for shortlisted issues to keep output small.
60+
- If `OTDASHBOARD_URL` or `OTDASHBOARD_TOKEN` is missing, ask the user to set the missing variable(s). Do not try to set them yourself
61+
- Do not echo the `OTDASHBOARD_TOKEN` variable and do not leak it anywhere.
62+
- Prefer `--json` for Jira issue fetches.

scripts/rota_ci_failures.py

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
#!/usr/bin/python
2+
#
3+
# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
4+
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
5+
#
6+
# The Universal Permissive License (UPL), Version 1.0
7+
#
8+
# Subject to the condition set forth below, permission is hereby granted to any
9+
# person obtaining a copy of this software, associated documentation and/or
10+
# data (collectively the "Software"), free of charge and under any and all
11+
# copyright rights in the Software, and any and all patent rights owned or
12+
# freely licensable by each licensor hereunder covering either (i) the
13+
# unmodified Software as contributed to or provided by such licensor, or (ii)
14+
# the Larger Works (as defined below), to deal in both
15+
#
16+
# (a) the Software, and
17+
#
18+
# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
19+
# one is included with the Software each a "Larger Work" to which the Software
20+
# is contributed by such licensors),
21+
#
22+
# without restriction, including without limitation the rights to copy, create
23+
# derivative works of, display, perform, and distribute the Software and make,
24+
# use, sell, offer for sale, import, export, have made, and have sold the
25+
# Software and the Larger Work(s), and to sublicense the foregoing rights on
26+
# either these or other terms.
27+
#
28+
# This license is subject to the following condition:
29+
#
30+
# The above copyright notice and either this complete permission notice or at a
31+
# minimum a reference to the UPL must be included in all copies or substantial
32+
# portions of the Software.
33+
#
34+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
35+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
36+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
37+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
38+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
39+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
40+
# SOFTWARE.
41+
42+
from __future__ import annotations
43+
44+
import json
45+
import os
46+
import sys
47+
import urllib.error
48+
import urllib.parse
49+
import urllib.request
50+
from collections.abc import Iterable
51+
from dataclasses import dataclass
52+
from datetime import UTC, datetime
53+
from typing import Any
54+
55+
API_PATH = "/api/periodic-jobs/GraalVM"
56+
REPO = "graalpython"
57+
BRANCH = "master"
58+
TARGETS = ["post-merge", "daily", "weekly", "monthly"]
59+
60+
61+
@dataclass(frozen=True)
62+
class FailureRow:
63+
target: str
64+
job_name: str
65+
last_successful_run: str
66+
jira_ids: str
67+
log_url: str
68+
69+
70+
class DashboardError(RuntimeError):
71+
pass
72+
73+
74+
def get_dashboard_token() -> str:
75+
token = os.environ.get("OTDASHBOARD_TOKEN")
76+
if token:
77+
return token
78+
raise DashboardError("OTDASHBOARD_TOKEN environment variable is not set.")
79+
80+
81+
def get_api_base() -> str:
82+
dashboard_url = os.environ.get("OTDASHBOARD_URL")
83+
if dashboard_url:
84+
return f"{dashboard_url.rstrip('/')}{API_PATH}"
85+
raise DashboardError("OTDASHBOARD_URL environment variable is not set.")
86+
87+
88+
def fetch_json(method: str, url: str, token: str, payload: dict[str, Any] | None = None) -> Any:
89+
data = None
90+
headers = {
91+
"Accept": "application/json",
92+
"Authorization": f"Bearer {token}",
93+
}
94+
if payload is not None:
95+
data = json.dumps(payload).encode("utf-8")
96+
headers["Content-Type"] = "application/json"
97+
request = urllib.request.Request(url, data=data, headers=headers, method=method)
98+
try:
99+
with urllib.request.urlopen(request, timeout=30) as response:
100+
return json.load(response)
101+
except urllib.error.HTTPError as exc:
102+
body = exc.read().decode("utf-8", "replace").strip()
103+
message = body or exc.reason
104+
raise DashboardError(f"Dashboard API request failed with HTTP {exc.code}: {message}") from exc
105+
except urllib.error.URLError as exc:
106+
raise DashboardError(f"Could not reach dashboard API: {exc.reason}") from exc
107+
except json.JSONDecodeError as exc:
108+
raise DashboardError(f"Dashboard API returned invalid JSON for {url}") from exc
109+
110+
111+
def get_latest_runs(api_base: str, token: str) -> dict[str, dict[str, Any]]:
112+
payload = {
113+
"repo": REPO,
114+
"branch": BRANCH,
115+
"targets": TARGETS,
116+
}
117+
result = fetch_json("POST", f"{api_base}/latest", token, payload)
118+
if not isinstance(result, dict):
119+
raise DashboardError("Dashboard API returned an unexpected payload for the latest runs.")
120+
121+
latest_runs: dict[str, dict[str, Any]] = {}
122+
for target in TARGETS:
123+
runs = result.get(target)
124+
if isinstance(runs, list) and runs and isinstance(runs[0], dict):
125+
latest_runs[target] = runs[0]
126+
return latest_runs
127+
128+
129+
def get_failed_jobs(api_base: str, token: str, run_id: str, target: str) -> list[dict[str, Any]]:
130+
params = urllib.parse.urlencode({
131+
"id": run_id,
132+
"status": "failed",
133+
"target": target,
134+
})
135+
url = f"{api_base}/jobs?{params}"
136+
result = fetch_json("GET", url, token)
137+
if not isinstance(result, list):
138+
raise DashboardError(f"Dashboard API returned an unexpected payload for failed jobs in target {target}.")
139+
return [job for job in result if isinstance(job, dict)]
140+
141+
142+
def format_timestamp_ms(timestamp_ms: Any) -> str:
143+
if not isinstance(timestamp_ms, int | float):
144+
return "-"
145+
return datetime.fromtimestamp(timestamp_ms / 1000, tz=UTC).strftime("%Y-%m-%d %H:%M:%S UTC")
146+
147+
148+
def format_jira_ids(tickets: Any) -> str:
149+
if not isinstance(tickets, Iterable) or isinstance(tickets, str | bytes):
150+
return "-"
151+
ticket_ids: list[str] = []
152+
for ticket in tickets:
153+
if isinstance(ticket, dict):
154+
ticket_id = ticket.get("ticketId")
155+
if isinstance(ticket_id, str) and ticket_id not in ticket_ids:
156+
ticket_ids.append(ticket_id)
157+
return ", ".join(ticket_ids) if ticket_ids else "-"
158+
159+
160+
def get_nested_timestamp(job: dict[str, Any]) -> Any:
161+
last_successful = job.get("lastSuccessful")
162+
if isinstance(last_successful, dict):
163+
return last_successful.get("run")
164+
return None
165+
166+
167+
def get_string(value: Any) -> str:
168+
return value if isinstance(value, str) and value else "-"
169+
170+
171+
def build_rows(api_base: str, token: str) -> list[FailureRow]:
172+
rows: list[FailureRow] = []
173+
latest_runs = get_latest_runs(api_base, token)
174+
for target in TARGETS:
175+
latest_run = latest_runs.get(target)
176+
if not latest_run:
177+
continue
178+
failed_count = latest_run.get("failed", 0)
179+
run_id = latest_run.get("id")
180+
if not isinstance(failed_count, int) or failed_count <= 0 or not isinstance(run_id, str):
181+
continue
182+
183+
for job in get_failed_jobs(api_base, token, run_id, target):
184+
rows.append(FailureRow(
185+
target=target,
186+
job_name=get_string(job.get("jobName")),
187+
last_successful_run=format_timestamp_ms(get_nested_timestamp(job)),
188+
jira_ids=format_jira_ids(job.get("tickets")),
189+
log_url=get_string(job.get("url")),
190+
))
191+
192+
return rows
193+
194+
195+
def print_table(rows: list[FailureRow]) -> None:
196+
if not rows:
197+
print("No failed jobs found in the latest periodic CI runs.")
198+
return
199+
200+
headers = ("target", "job name", "last successful run", "jira ID(s)", "log URL")
201+
values = [headers]
202+
values.extend((row.target, row.job_name, row.last_successful_run, row.jira_ids, row.log_url) for row in rows)
203+
widths = [max(len(str(row[column])) for row in values) for column in range(len(headers))]
204+
205+
def render(columns: tuple[str, ...]) -> str:
206+
return " | ".join(value.ljust(widths[index]) for index, value in enumerate(columns))
207+
208+
separator = "-+-".join("-" * width for width in widths)
209+
print(render(headers))
210+
print(separator)
211+
for row in rows:
212+
print(render((row.target, row.job_name, row.last_successful_run, row.jira_ids, row.log_url)))
213+
214+
215+
def main() -> int:
216+
try:
217+
api_base = get_api_base()
218+
token = get_dashboard_token()
219+
rows = build_rows(api_base, token)
220+
print_table(rows)
221+
except DashboardError as exc:
222+
print(f"error: {exc}", file=sys.stderr)
223+
return 1
224+
return 0
225+
226+
227+
if __name__ == "__main__":
228+
raise SystemExit(main())

0 commit comments

Comments
 (0)