ci(feat): refactor error handling by joshmarinacci · Pull Request #25610 · hiero-ledger/hiero-consensus-node

joshmarinacci · 2026-05-22T21:45:38Z

Description:

Create a common set-failure-mode workflow that can be used to set the failure mode variable of a single task, or aggregate a bunch of child tasks.

Related issue(s):

Partially addresses #25509 and #25513 and #25470

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

trunk-io · 2026-05-22T21:45:47Z

Merging to main in this repository is managed by Trunk.

To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

lfdt-bot · 2026-05-22T21:46:02Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

Copilot

Pull request overview

This PR centralizes CI failure-mode calculation into a shared workflow support script and refactors several zxc-* workflows to use it, adding failed-tests aggregation where applicable.

Changes:

Adds .github/workflows/support/scripts/set-failure-mode.sh for step-outcome and job-aggregate failure classification.
Refactors multiple CI workflows to pass step/job status arrays into the shared script.
Updates MATS downstream categorization to treat the new cancelled failure mode as environment-related.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`.github/workflows/support/scripts/set-failure-mode.sh`	Adds shared failure-mode and failed-tests output logic.
`.github/workflows/zxc-compile-and-spotless-check.yaml`	Replaces inline failure-mode logic with script inputs.
`.github/workflows/zxc-dependency-module-check.yaml`	Replaces inline failure-mode logic with script inputs.
`.github/workflows/zxc-execute-unit-tests.yaml`	Refactors unit-test failure-mode output to use shared script.
`.github/workflows/zxc-execute-hapi-tests.yaml`	Refactors HAPI aggregate failure-mode logic and adds failed-tests job output.
`.github/workflows/zxc-execute-otter-tests.yaml`	Refactors Otter aggregate failure-mode logic and adds failed-tests job output.
`.github/workflows/zxc-mats-tests.yaml`	Refactors MATS aggregate failure-mode/failed-tests logic.
`.github/workflows/zxc-verify-gradle-build-determinism.yaml`	Refactors Gradle determinism aggregate failure-mode logic.
`.github/workflows/zxc-verify-docker-build-determinism.yaml`	Refactors Docker determinism aggregate failure-mode logic.
`.github/workflows/zxc-xts-tests.yaml`	Refactors XTS aggregate failure-mode/failed-tests logic, including panel jobs.
`.github/workflows/node-flow-build-application.yaml`	Maps MATS `cancelled` failure mode to environment category.

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

andrewb1269 · 2026-05-28T19:33:29Z

We are trying to provide a reason for why XTS workflows are cancelled to our Slack channels. I suggest we provide it as a separate variable, rather than trying to include it into the failure_mode variable. If we provide a separate variable, we can populate that variable only when there is a case to check.

andrewb1269 · 2026-05-28T19:16:40Z

+if has_failure "${test_arr[@]}"; then
+  failure_mode="test"
+fi
+if has_failure "${workflow_arr[@]}"; then
+  failure_mode="workflow"
+fi


If there is a test failure and a workflow failure, then the test failure is overwritten by the workflow failure. Suggest changing to this to elif so if we have a test failure we do not worry about the workflow failure(s).

andrewb1269 · 2026-05-28T19:24:28Z

+      workflow) failure_mode="workflow" ;;
+      test) [[ "$failure_mode" != "workflow" ]] && failure_mode="test" ;;


This will overwrite the failure_mode to workflow and hide potential test failure types. We want to prioritize test failures over workflow failures.

andrewb1269 · 2026-05-28T19:27:34Z

+    esac
+    failed_tests="${failed_tests:+${failed_tests}, }${name}"
+  elif [[ "$status" == "cancelled" ]]; then
+    [[ "$failure_mode" == "none" ]] && failure_mode="cancelled"


failure_mode can be workflow, test, or none. It should stay as none since there was not a failure, it was only a cancelled workflow.

andrewb1269 · 2026-05-28T19:28:19Z

+    failed_tests="${failed_tests:+${failed_tests}, }${name}"
+  elif [[ "$status" == "cancelled" ]]; then
+    [[ "$failure_mode" == "none" ]] && failure_mode="cancelled"
+    failed_tests="${failed_tests:+${failed_tests}, }${name} (cancelled)"


We don't know if it was a failed test or not, so we should not add it to the failed_tests list.

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

andrewb1269 · 2026-06-01T18:08:07Z

The remaining comments are regarding:

Cancelled tests being processed vs. not being processed. I'd propose we do NOT use cancelled tests in this PR and can discuss for a follow-on PR.
The priority of test failures vs. workflow failures. I think the order should be: test > workflow > none, then we will add cancelled in a future PR giving us test > workflow > cancelled > none

joshmarinacci added 2 commits May 22, 2026 10:53

add error analysis

ab52990

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

add set-failure-mode

b404cc0

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

joshmarinacci requested a review from a team as a code owner May 22, 2026 21:45

joshmarinacci requested a review from nathanklick May 22, 2026 21:45

joshmarinacci self-assigned this May 22, 2026

joshmarinacci added 2 commits May 22, 2026 15:10

fix path to github action

9c7bdc5

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

remove unneeded files

069cda4

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

joshmarinacci force-pushed the failure-refactor branch from 71410a3 to 069cda4 Compare May 26, 2026 15:51

joshmarinacci added 9 commits May 26, 2026 09:29

consolidate set-failure-mode

7cc28c2

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

consolidate more set-failure-mode

9fb8f47

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

fix needs

38274d9

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

spotless

5b52980

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

switch from callable workflow to a bash script

9d25337

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

cleanup

63efec1

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

fixes

afe9f23

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

minimal diff

a08ef9c

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

minimal diff

04505d0

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

This comment has been minimized.

Sign in to view

fix broken checkout

e30b4f9

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

andrewb1269 requested review from andrewb1269 and Copilot and removed request for nathanklick May 28, 2026 14:46

Copilot started reviewing on behalf of andrewb1269 May 28, 2026 14:47 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread .github/workflows/zxc-xts-tests.yaml

Comment thread .github/workflows/zxc-execute-unit-tests.yaml Outdated

Comment thread .github/workflows/support/scripts/set-failure-mode.sh

Comment thread .github/workflows/zxc-verify-docker-build-determinism.yaml Outdated

joshmarinacci added 3 commits May 28, 2026 10:31

checkout before running the script

28a6c40

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

fix version of the github job

93a800c

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

fix needs to steps

a1ee1fc

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

andrewb1269 requested changes May 28, 2026

View reviewed changes

joshmarinacci added 3 commits June 1, 2026 08:54

add comments

d0a74bb

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

add comments

1e77cbc

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

add more comments

cf90df8

Signed-off-by: Josh Marinacci <joshua@marinacci.org>

		workflow) failure_mode="workflow" ;;
		test) [[ "$failure_mode" != "workflow" ]] && failure_mode="test" ;;

Conversation

joshmarinacci commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented May 22, 2026

Uh oh!

lfdt-bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewb1269 commented May 28, 2026

Uh oh!

andrewb1269 May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewb1269 May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewb1269 May 28, 2026

Choose a reason for hiding this comment

Uh oh!

andrewb1269 May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrewb1269 commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshmarinacci commented May 22, 2026 •

edited

Loading

lfdt-bot commented May 22, 2026 •

edited

Loading