Refactor IssueHandler from supervisor to issue_verification_agent by martinky82 · Pull Request #473 · packit/ai-workflows

martinky82 · 2026-05-12T11:06:59Z

TODO:

Write new tests or update the old ones to cover new functionality.
Update doc-strings where appropriate.
Update or write new documentation in packit/packit.dev.
‹fill in›

Fixes

Related to

Merge before/after

RELEASE NOTES BEGIN

Packit now supports automatic ordering of ☕ after all checks pass.

RELEASE NOTES END

gemini-code-assist

Code Review

This pull request introduces the Issue Verification Agent, migrating the post-fix lifecycle management from the supervisor to a dedicated BeeAI workflow. Key additions include modules for baseline test reproduction, XUnit comparison, and a testing analyst agent, along with several new tools for interacting with Jira, GitLab, the Errata Tool, and Testing Farm. Feedback highlights a critical bug in the Jira attachment tool call involving incorrect data structures and encoding. Performance improvements were also requested to wrap blocking synchronous calls in threads within asynchronous methods, and a simplification was suggested for unreachable code in the GitLab MR search logic.

TomasTomecek

wow, this is massive, thank you for working on this

are you also planning to drop the former supervisor code?

btw this definitely needs rebase against upstream/main

martinky82 · 2026-05-12T12:27:13Z

After the errata agent is moved out of supervisor, the supervisor will be dropped altogether of course. I reckon it's on the safe side to keep the code in there for the moment.

Yeah, some stuff landed since I started working on it so rebase it needs.

Move the IssueHandler workflow from ymir/supervisor/ into a standalone BeeAI Framework agent at ymir/agents/issue_verification_agent.py, following the pattern established by preliminary_testing_agent. This includes: - New agent: issue_verification_agent with Workflow-based state machine - New sub-agent: testing_analyst for AI-driven test result analysis - New MCP tools: errata (GetErratum, GetErratumBuildNvr), testing_farm (GetTestingFarmRequest, ReproduceTestingFarmRequest), gitlab (SearchGitlabProjectMrs), jira (UpdateJiraComment, AddJiraAttachments, GetJiraAttachment) - New unprivileged tools: read_logfile, read_readme, search_resultsdb, analyze_ewa_testrun, read_attachment - Supporting modules: baseline_tests, compare_xunit, qe_data - Shared types added to ymir/common/models.py - New skill definition for issue_verification The supervisor's ErratumHandler remains unchanged for a follow-up. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

martinky82 · 2026-05-12T13:19:48Z

I rebased the PR and now I have the pre-commit installed as well, so it should be in better shape than the first attempt.

The agent only checked customfield_10418 for the errata link, but some issues (e.g. those handled by infraserv teams directly) store the errata link in customfield_10626 instead. Fall back to that field when the primary one is null. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

TomasTomecek

please address the bug with the issue_key, the other 2 comments are mostly cosmetic

TomasTomecek · 2026-05-13T11:40:30Z

+            return JSONToolOutput(result=results)
+
+        except Exception as e:
+            from beeai_framework.tools import ToolError


this shouldn't be here

Could you, please, elaborate?

Hopefully OK this time

Inlines the function and its regex constants from ymir/supervisor/ewa_utils.py, removing the cross-module dependency on the supervisor package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

TomasKorbar · 2026-05-15T09:27:23Z

Additional note, please implement makefile target for this so it can be triggered as the other agents. For example like we do with triage make run-triage-agent-standalone DRY_RUN=true

The get_jira_pull_requests and get_jira_dev_status tools were returning bare lists, but MCP requires structuredContent to be dictionaries. This caused validation errors when these tools returned results (including empty lists). Changes: - Wrap return values in dictionaries with descriptive keys (pull_requests, commits) - Update return type annotations to match new structure - Update preliminary_testing_agent to extract list from dictionary Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Move the IssueHandler workflow from ymir/supervisor/ into a standalone BeeAI Framework agent at ymir/agents/issue_verification_agent.py, following the pattern established by preliminary_testing_agent. This includes: - New agent: issue_verification_agent with Workflow-based state machine - New sub-agent: testing_analyst for AI-driven test result analysis - New MCP tools: errata (GetErratum, GetErratumBuildNvr), testing_farm (GetTestingFarmRequest, ReproduceTestingFarmRequest), gitlab (SearchGitlabProjectMrs), jira (UpdateJiraComment, AddJiraAttachments, GetJiraAttachment) - New unprivileged tools: read_logfile, read_readme, search_resultsdb, analyze_ewa_testrun, read_attachment - Supporting modules: baseline_tests, compare_xunit, qe_data - Shared types added to ymir/common/models.py - New skill definition for issue_verification The supervisor's ErratumHandler remains unchanged for a follow-up. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The agent only checked customfield_10418 for the errata link, but some issues (e.g. those handled by infraserv teams directly) store the errata link in customfield_10626 instead. Fall back to that field when the primary one is null. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Hopefully OK this time

Inlines the function and its regex constants from ymir/supervisor/ewa_utils.py, removing the cross-module dependency on the supervisor package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The get_jira_pull_requests and get_jira_dev_status tools were returning bare lists, but MCP requires structuredContent to be dictionaries. This caused validation errors when these tools returned results (including empty lists). Changes: - Wrap return values in dictionaries with descriptive keys (pull_requests, commits) - Update return type annotations to match new structure - Update preliminary_testing_agent to extract list from dictionary Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

Convert the stub SKILL.md into a comprehensive skill definition matching the pattern used by preliminary_testing and triage skills. Includes all workflow steps, tool listings, constants, attention template, and output schema with examples. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Move baseline_tests.py and compare_xunit.py to ymir/agents/utilities/ subpackage to separate utility code from agent modules - Extract MERGE_CHECK_DELAY and ERRATA_WAIT_DELAY constants in issue_verification_agent.py - Convert testing_analyst from deprecated ToolCallingAgent to ReasoningAgent with proper config (get_chat_model, is_reasoning_enabled, get_tool_call_checker_config) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move the testing analyst sub-agent from a separate module directly into issue_verification_agent.py, consistent with how other workflows embed their sub-agents. Rename schemas and templates with TestingAnalyst/TESTING_ANALYST_ prefixes to avoid collisions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use the get_maintainer_rules MCP tool to fetch package-specific testing guidelines from AGENTS.md instead of the hardcoded jotnar-qe-data repository. Teams can now specify test location info in their rules repository. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Wrap synchronous network calls in asyncio.to_thread in errata.py (GetErratumTool, GetErratumBuildNvrTool) and analyze_ewa_testrun.py to avoid blocking the event loop - Remove redundant inline aiohttp import in jira.py, use the existing top-level import Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ean up tools Move JIRA attachment reading to the privileged jira module by replacing ReadAttachmentTool with the existing GetJiraAttachmentTool. Also raise ToolError instead of ValueError in testing_farm.py and remove unnecessary type: ignore comments from unprivileged tools. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Security should be enforced at the infrastructure level via egress control, not by hardcoded URL patterns in tools. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove unnecessary type: ignore comments, update description to reference maintainer rules instead of removed TEST_LOCATION_INFO, and replace broad Exception catch with specific aiohttp.ClientError and ValueError raised as ToolError. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ervice Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

Co-authored-by: Tomas Tomecek <tomas@tomecek.net>

Use list.extend instead of loop append (PERF401) and fix two lines exceeding 110 character limit (E501). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

TomasTomecek

I used opus to review this PR and overall it's very well done there is just one internal API breakage:

Broken caller of `get_jira_dev_status` in `zstream_search.py`

The return type of GetJiraDevStatusTool and GetJiraPullRequestsTool changed from a bare list to a dict wrapper ({"commits": [...]} / {"pull_requests": [...]}). preliminary_testing_agent.py was updated to handle
this, but zstream_search.py was missed.

In ymir/tools/privileged/zstream_search.py around line 235-244:

dev_status = await run_tool(GetJiraDevStatusTool(), issue_key=issue_key)
commits = json.loads(dev_status) if isinstance(dev_status, str) else dev_status
# ...
commit_urls = [_get_patch_url(c["url"]) for c in commits if c.get("url")]

After this change, commits will be {"commits": [...]} instead of a list, so iterating it will iterate over dict keys ("commits") rather than commit objects, causing a KeyError.

The test mocks in ymir/agents/tests/unit/test_zstream_search.py also still return bare lists, so the tests pass but don't catch this.

Otherwise LGTM, thank you for working on this!! Looking fwd to trying this out.

GetJiraDevStatusTool now returns {"commits": [...]} instead of a bare list. Update zstream_search.py to extract the commits list and fix test mocks to match the new format. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

martinky82 · 2026-05-20T14:37:51Z

I used opus to review this PR and overall it's very well done there is just one internal API breakage:

Broken caller of get_jira_dev_status in zstream_search.py

Good catch, thanks! Fixed in the last commit.

martinky82 · 2026-05-20T14:38:29Z

Additional note, please implement makefile target for this so it can be triggered as the other agents. For example like we do with triage make run-triage-agent-standalone DRY_RUN=true

Makefile target is implemented, thanks for the hint.

gemini-code-assist Bot reviewed May 12, 2026

View reviewed changes

Comment thread ymir/agents/utilities/baseline_tests.py

Comment thread ymir/tools/privileged/errata.py

Comment thread ymir/tools/unprivileged/analyze_ewa_testrun.py Outdated

Comment thread ymir/agents/issue_verification_agent.py

TomasTomecek reviewed May 12, 2026

View reviewed changes

martinky82 force-pushed the main branch from 63590e2 to 3c22373 Compare May 12, 2026 13:10

martinky82 force-pushed the main branch from 024e20f to 03eab1a Compare May 12, 2026 13:18

TomasTomecek requested changes May 13, 2026

View reviewed changes

TomasKorbar self-requested a review May 14, 2026 09:19

martinky82 and others added 3 commits May 14, 2026 13:20

fix the issues found in review

c8af322

repeated fix of issues found bby review

876637b

Hopefully OK this time

Move get_tcms_run_details into analyze_ewa_testrun tool

090327d

Inlines the function and its regex constants from ymir/supervisor/ewa_utils.py, removing the cross-module dependency on the supervisor package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

TomasKorbar requested changes May 15, 2026

View reviewed changes

martinky82 and others added 15 commits May 20, 2026 00:57

fix double attachments: error in ymir/agents/baseline_tests.py

be6d4f7

fix the issues found in review

dfd8fff

repeated fix of issues found bby review

0ff72cb

Hopefully OK this time

Move get_tcms_run_details into analyze_ewa_testrun tool

ab819a3

Inlines the function and its regex constants from ymir/supervisor/ewa_utils.py, removing the cross-module dependency on the supervisor package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix double attachments: error in ymir/agents/baseline_tests.py

519c777

ai-workflows: fix merge conflict

9ca06b0

[pre-commit.ci] auto fixes from pre-commit.com hooks

8500027

for more information, see https://pre-commit.ci

delete unreachable code

0aefe4e

martinky82 and others added 14 commits May 20, 2026 11:05

Rename JotnarTag to YmirTag in common models

d155bc7

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'packit:main' into main

41d3795

Merge branch 'main' of github.com:martinky82/ai-workflows

e0eab0f

Remove URL allowlisting from read_logfile tool

dfe5c26

Security should be enforced at the infrastructure level via egress control, not by hardcoded URL patterns in tools. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add issue-verification-agent standalone Makefile target and compose s…

c034dc4

…ervice Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

52c5457

for more information, see https://pre-commit.ci

Update ymir/agents/issue_verification_agent.py

0c4e343

Co-authored-by: Tomas Tomecek <tomas@tomecek.net>

Fix linting errors in issue_verification_agent

41583ab

Use list.extend instead of loop append (PERF401) and fix two lines exceeding 110 character limit (E501). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'main' of github.com:martinky82/ai-workflows

812ca97

[pre-commit.ci] auto fixes from pre-commit.com hooks

9c4d0e8

for more information, see https://pre-commit.ci

TomasTomecek requested changes May 20, 2026

View reviewed changes

martinky82 and others added 3 commits May 20, 2026 16:35

Merge branch 'main' of github.com:martinky82/ai-workflows

b779712

[pre-commit.ci] auto fixes from pre-commit.com hooks

2fae077

for more information, see https://pre-commit.ci

Conversation

martinky82 commented May 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomasTomecek left a comment

Choose a reason for hiding this comment

Uh oh!

martinky82 commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martinky82 commented May 12, 2026

Uh oh!

TomasTomecek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TomasTomecek May 13, 2026

Choose a reason for hiding this comment

Uh oh!

martinky82 May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomasKorbar commented May 15, 2026

Uh oh!

TomasTomecek left a comment

Choose a reason for hiding this comment

Broken caller of get_jira_dev_status in zstream_search.py

Uh oh!

martinky82 commented May 20, 2026

Broken caller of get_jira_dev_status in zstream_search.py

Uh oh!

martinky82 commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martinky82 commented May 12, 2026 •

edited

Loading

Broken caller of `get_jira_dev_status` in `zstream_search.py`

Broken caller of `get_jira_dev_status` in `zstream_search.py`