[AIROCMLIR-597][CI] Extract Jenkinsfile helpers into helpers/*.groovy and load them in a Bootstrap stage

[bogdan-petkovic]

Motivation

The primary mlir/utils/jenkins/Jenkinsfile has grown to ~2000 lines, the majority of which are helper functions defined at the top of the file. This makes the actual pipeline definition hard to read and to navigate, and forces the same helpers to be duplicated across Jenkinsfile.downstream and Jenkinsfile.release (see the // ON CHANGING THESE, ALSO CHANGE Jenkinsfile.downstream markers). Extracting the helpers into a shared location is the first step toward eliminating that duplication.
Two earlier attempts addressed this problem but neither was merged:

• #2092 — CI: Refactor Jenkinsfile helpers into implicit Shared Library — split the helpers into multiple files under a vars/ directory at the repo root. Clean per-domain separation, but the implicit-Shared-Library mechanism requires the Jenkins controller to be configured with a Global Pipeline Library.
• #2276 — [AIROCMLIR-597][CI] Create PipelineHelpers and load them in Jenkinsfile via Bootstrap stage — collapsed every helper into a single PipelineHelpers.groovy and loaded it from a Bootstrap stage. Fully self-contained, but consolidates ~2000 lines into one file, which keeps the original navigability problem.
  This PR combines the strengths of both: it adopts CI: Refactor Jenkinsfile helpers into implicit Shared Library #2092's per-domain file split (helpers grouped by purpose into separate files) and [AIROCMLIR-597][CI] Create PipelineHelpers and load them in Jenkinsfile via Bootstrap stage #2276's load()-from-Bootstrap loading mechanism.

Technical Details

• New mlir/utils/jenkins/helpers/ directory with six files: scmUtils, nodeUtils, buildUtils, testUtils, reportUtils, ciLogic — 37 helpers total.
• New Bootstrap stage in Jenkinsfile does checkout scm, loads each helper file, and wires cross-helper handles via @Field
  (nodeUtils.scmUtils, testUtils.nodeUtils, testUtils.buildUtils).
• All helper call sites in Jenkinsfile are namespaced (e.g. ciLogic.setHeartbeat()).
• Jenkinsfile: 2012 → 1099 lines.
• All extracted helpers are identical to develop except the cross-helper namespace prefixes and one deliberate shell bug fix in testUtils.collectCoverageData, per Copilot review on PR [AIROCMLIR-597][CI] Create PipelineHelpers and load them in Jenkinsfile via Bootstrap stage #2276 ([AIROCMLIR-597][CI] Create PipelineHelpers and load them in Jenkinsfile via Bootstrap stage #2276 (review)):

  -       if [ `df --output=avail -k . | tail -1l` -lt 153600000 ]; then
  +       if [ `df --output=avail -k . | tail -n 1` -lt 153600000 ]; then
  

• Jenkinsfile.downstream and Jenkinsfile.release are unchanged here; a follow-up will migrate them so the // ON CHANGING THESE, ALSO CHANGE Jenkinsfile.downstream duplication can be removed.

Test Plan

• Full CI matrix on this PR before marking it ready for review:
  • PR CI — default codepath
  • PR CI — mfma
  • PR CI — navi21
  • PR CI — navi3x
  • PR CI — navi4x
  • PR CI — vanilla
  • PR CI — gfx950
  • Nightly
  • Weekly

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[dorde-antic]

lgtm

[Copilot]

Pull request overview

This PR refactors the main CI pipeline definition by extracting Jenkinsfile helper logic into per-domain Groovy scripts under mlir/utils/jenkins/helpers/, then loading and wiring those helpers in a new Bootstrap stage. This reduces the size/complexity of the primary Jenkinsfile and centralizes shared helper logic to support future de-duplication with downstream/release pipelines.

Changes:

• Added a Bootstrap stage that checks out SCM, load()s helper scripts, and wires cross-helper dependencies.
• Moved helper methods out of mlir/utils/jenkins/Jenkinsfile into six domain-focused helper files.
• Updated Jenkinsfile call sites to use namespaced helper handles (e.g., ciLogic.*, nodeUtils.*, scmUtils.*, etc.).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
mlir/utils/jenkins/Jenkinsfile	Removes inlined helpers, adds Bootstrap load/wiring, and updates call sites to namespaced helper objects.
mlir/utils/jenkins/helpers/scmUtils.groovy	Extracts SCM-related helpers (git health check, robust checkout).
mlir/utils/jenkins/helpers/nodeUtils.groovy	Extracts node health/Docker discovery logic and withHealthyNode() harness.
mlir/utils/jenkins/helpers/buildUtils.groovy	Extracts build helpers for rocMLIR + CK/MIGraphX.
mlir/utils/jenkins/helpers/testUtils.groovy	Extracts test/sweep/static-check/coverage helpers (includes the tail -n 1 fix).
mlir/utils/jenkins/helpers/reportUtils.groovy	Extracts report publication and perfDB archival helpers.
mlir/utils/jenkins/helpers/ciLogic.groovy	Extracts CI flow logic (labels/gating/splitting/failure classification/Teams notifications).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dorde-antic]

lgtm, finally a cleaner Jenkinsfile :)
wait for all types of CI runs to be successful before merging

[rocmlir-pr-reviewer]

Reviewed the Jenkinsfile + 6 new helper files (1 Jenkinsfile, 6 helpers/*.groovy). The refactor cleanly extracts helpers per domain and wires them via load() in the new Bootstrap stage; all call sites in the Jenkinsfile and inside the extracted ciLogic matrix-row orchestrators use the correct single-prefix form (e.g. buildUtils.buildProject(...), not buildUtils.buildUtils.buildProject(...)), and reportUtils.archivePerfDB() is invoked from a script {} block. The two prior Copilot findings on PR #2360 appear already addressed in this HEAD. CI Bootstrap, Set System Property, Kill old PR builds, and Build and Test all passed. Posted 0 inline comments (0 critical, 0 major, 0 minor). Verdict: APPROVE.

[rocmlir-pr-reviewer]

Reviewed 7 files (Jenkinsfile + 6 helper modules under mlir/utils/jenkins/helpers/). Posted 0 inline comments (0 critical, 0 major, 0 minor). Verdict: APPROVE. This is a clean mechanical extraction of ~900 lines of helpers from Jenkinsfile (2012 -> 333 lines) into six domain-grouped .groovy files loaded by a new Bootstrap stage. Cross-helper handles are wired in Bootstrap (nodeUtils.scmUtils, testUtils.nodeUtils/buildUtils, and ciLogic. fan-out), and every call site in Jenkinsfile is correctly namespaced (e.g. ciLogic.setHeartbeat(), reportUtils.archivePerfDB() inside a script {} block). The two earlier Copilot findings on commit 947e828 (a buildUtils.buildUtils.buildProject(...) double-prefix typo and a missing script {} wrapper around reportUtils.archivePerfDB()) are both fixed in the current head (ec97752). The one deliberate behavioral change called out in the description (tail -1l -> tail -n 1 in testUtils.collectCoverageData) is correctly applied at testUtils.groovy:117. Jenkinsfile.downstream / Jenkinsfile.release migration is deferred to a follow-up as stated. Note for CI state: at review time the heavy Jenkins matrix stages (MIGraphX, Tune, Benchmark, Code coverage, Parameter sweeps, Archive weekly tuning perfDB, ml-ci-internal) are still PENDING -- the description shows the PR-CI matrix has been re-run green, but the weekly box remains unchecked and final green from the in-flight run should be confirmed before merge.

[umangyadav]

Looks good. Make sure to test this out in Jenkins using PR CI, Nightly CI and weekly CI triggers

[rocmlir-pr-reviewer]

Verdict: APPROVE -- submitted as COMMENT (automated reviews are advisory)  ·  Findings: 0 (0 Critical, 0 Major, 0 Minor)

---

Scope

Mechanical refactor that extracts ~900 lines of Groovy helper methods out of mlir/utils/jenkins/Jenkinsfile (2012 → 1099 lines) into six per-domain files under mlir/utils/jenkins/helpers/ (scmUtils, nodeUtils, buildUtils, testUtils, reportUtils, ciLogic), loaded from a new Bootstrap stage via load() and consumed through namespaced handles wired together with @Field cross-helper references.

Findings

No blocking issues found.

Notes

• The two prior Copilot review findings appear addressed in the head commit:
  • The buildUtils.buildUtils.buildProject(...) typo flagged at the old Prepare Performance Scripts site is no longer present; ciLogic.runParameterSweepsMatrixRow now invokes buildUtils.buildProject('check-rocmlir-build-only ci-performance-scripts', '') with a single-level handle.
  • reportUtils.archivePerfDB() is now wrapped in script { ... } inside the Archive weekly tuning perfDB stage's steps block (Jenkinsfile lines 239–241), so the Groovy method call is no longer issued directly to Declarative steps.
• Cross-helper handles are declared with @Field in each consumer file (nodeUtils.scmUtils, testUtils.{nodeUtils,buildUtils}, ciLogic.{scmUtils,nodeUtils,buildUtils,testUtils,reportUtils}) and all of them are populated in the Bootstrap stage — no dangling references.
• Bare setHeartbeat() calls inside ciLogic.groovy (e.g. lines 652, 718) resolve correctly because they target methods on the same loaded script object.
• The intentional tail -1l → tail -n 1 shell fix in testUtils.collectCoverageData is present and matches the PR description.
• Pre-existing issues (out of scope): ciLogic.shouldRunBuildAndTest(...) falls off the end without an explicit return false when params.codepath is neither "default" nor equal to codepath; this implicit-null behavior was inherited from develop. ciLogic.setHeartbeat() wraps its body in a script { } step (a Declarative-only construct), also inherited verbatim. Neither was introduced by this PR; consider a follow-up cleanup.
• The Bootstrap stage pins an mlir-labelled executor for checkout scm + load; the inline comment explains the rationale and the per-matrix stages still allocate their own agents through nodeUtils.withHealthyNode.

CI status

Python format and lint checks, Python performance script tests, and detect are green. C/C++ premerge checks is still in-progress on the pinned SHA — not a failure.

[justinrosner]

Nice!