[DX-3522] Release testing by Tofel · Pull Request #21637 · smartcontractkit/chainlink

Tofel · 2026-03-23T16:45:37Z

trigger legacy, CCIP in-memory tests CRE tests and DF1 and CRE upgrade tests, when a new CL image is published
trigger nightly legacy tests from Docker Build instead of fixed schedule

Add action to resolve CL image to use with following characteristics:

Supports two input modes:
- Full image mode via chainlink_full_image (/: or /@sha256:).
- Composed ECR mode via chainlink_image_repo + tag fallback chain (chainlink_image_tag -> chainlink_version -> github.sha).

Enforces input safety/consistency:
- chainlink_full_image is mutually exclusive with chainlink_image_repo and chainlink_image_tag.
- Validates full-image format and fails fast with clear ::error:: messages on invalid combinations/format.
In composed mode, defaults repo to chainlink and builds:
- ${QA_AWS_ACCOUNT_NUMBER}.dkr.ecr.${QA_AWS_REGION}.amazonaws.com/:<resolved_tag>
Exposes the final value through action output resolved_image for downstream workflow steps.

Tested:

nightly with public image ✅
nightly with nightly image ✅
CRE smoke & regression in PR ✅
CRE smoke with public image ✅
CRE smoke with private image

…er nightly legacy tests from Docker Build instead of fixed schedule

github-actions · 2026-03-23T16:47:07Z

✅ No conflicts with other open PRs targeting develop

trunk-io · 2026-03-23T18:05:35Z

_{View Full Report ↗︎ ⋅ Docs}

Copilot

Pull request overview

Risk Rating: HIGH (changes to CI/CD workflows that gate releases and nightly testing)

This PR updates GitHub Actions workflows to run legacy system tests and CRE system/regression tests based on newly published Chainlink images (and shifts “nightly” execution to be driven by Docker Build / build-publish workflows rather than a fixed cron in the system-tests workflow itself).

Changes:

Convert (Nightly) System Tests to be reusable (workflow_call) / parameterized, and add logic to resolve the Chainlink image reference from inputs.
Update integration workflows to pass the new chainlink_image_repo input name to CRE reusable workflows.
Trigger legacy + CRE tests from image build/publish workflows (docker-build, build-publish) and enable calling integration-in-memory tests via workflow_call.

Scrupulous human review areas:

The new “Resolve Chainlink image” bash logic in system-tests-nightly.yml, cre-system-tests.yaml, and cre-regression-system-tests.yaml (input validation + image ref construction).
The new post-build test triggers in build-publish.yml and docker-build.yml (ensuring intended event coverage, permissions, and failure behavior).

Reviewer recommendations (per CODEOWNERS):

@smartcontractkit/devex-cicd (explicit owner for .github/workflows/build-publish.yml, and owner for /.github/**)
@smartcontractkit/devex-tooling and/or @smartcontractkit/core (owners for /.github/**)

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.github/workflows/system-tests-nightly.yml	Makes system tests reusable/parameterized and adds Chainlink image resolution logic.
.github/workflows/integration-tests.yml	Updates CRE reusable workflow input name from `ecr_name` to `chainlink_image_repo`.
.github/workflows/integration-in-memory-tests.yml	Adds `workflow_call` input and allows execution when called as a reusable workflow.
.github/workflows/docker-build.yml	Calls system tests workflow after image build on schedule/dispatch.
.github/workflows/cre-system-tests.yaml	Adds new image inputs and resolves Chainlink image before running CRE system tests.
.github/workflows/cre-regression-system-tests.yaml	Adds new image inputs and resolves Chainlink image before running CRE regression tests.
.github/workflows/build-publish.yml	Triggers CRE regression, legacy system tests, and CCIP in-memory tests after publish.

Copilot

Pull request overview

Copilot reviewed 14 out of 17 changed files in this pull request and generated 3 comments.

…rkflow

cl-sonarqube-production · 2026-03-26T08:47:58Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

erikburt · 2026-03-26T16:05:32Z

-        description: "The version of Chainlink repository to use for the tests."
-        default: "develop"
+        description: "The version of Chainlink repository to use for the tests. If empty, defaults to github.sha."
+        default: ""


Shouldn't this have a ${{ github.sha }}?

erikburt · 2026-03-26T16:06:57Z

        required: true
        type: string
        description: "The version of Chainlink repository to use for the tests."
        default: "develop"


Why does the workflow_call use develop but the workflow_dispatch use ${{ github.sha }}?

erikburt · 2026-03-26T16:08:20Z


+      - name: Authenticate to ECR (public)
+        id: login-ecr-public
+        uses: aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076 # v2.0.1


Update this to:

@183a1442edf41672e66566b7fc560e297a290896 # v2.1.1

erikburt · 2026-03-26T16:11:30Z

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+set -euo pipefail


Brief header comment would be nice

# Resolves the Chainlink Docker image to use based on environment variables. # Priority: # 1. If CHAINLINK_FULL_IMAGE is set, use it directly (must include registry and tag/digest). # 2. Otherwise, construct the image from: # - Repository: CHAINLINK_IMAGE_REPO (default: "chainlink") # - Tag: CHAINLINK_IMAGE_TAG or fallback to CHAINLINK_VERSION # - Registry: AWS ECR using AWS_ACCOUNT_NUMBER and AWS_REGION # Enforces mutual exclusivity between FULL_IMAGE and repo/tag inputs, and validates required fields.

erikburt · 2026-03-26T16:19:24Z

+  # Core release tests
+  # Smoke & regression tests
+  run-cre-system-tests:
+    name: Run CRE System Tests


Is there a requirement to having these inline? This seems fine for now, but I think it would make sense to have a post-build-publish.yml workflow that is either called by this one, or is dispatched using a workflow_run: [completed] event (https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run)

It would allow us to:

actually test the post-workflow steps

stop polluting release workflows with arbtirary testing jobs

This can be done later because it's not particularly easy to pass information between these jobs, so that would need to be figured out.

erikburt · 2026-03-26T16:25:21Z

@@ -1 +1 @@
 name: Upgrade Compatibility Test


From what I can tell, this workflow is not being executed/tested by this PR. It is only run on nightlies at the moment.

IMO it makes sense to update this workflow and the nightly workflow in a separate PR and let it bake to test that it's working before referencing it in our release workflow.

erikburt · 2026-03-26T16:26:46Z

  push:
    tags:
      - "*"


This already runs on tag pushes, so if we're adding it to be called from the release pipeline which also runs on tag pushes, it will run twice.

erikburt · 2026-03-26T16:27:01Z

+      contents: read
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4


erikburt · 2026-03-26T16:29:29Z

        uses: aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076 # v2.0.1
+        with:
+          registries: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}
+        env:
+          AWS_REGION: ${{ secrets.QA_AWS_REGION }}
+
+      - name: Authenticate to ECR (public)
+        id: login-ecr-public
+        uses: aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076 # v2.0.1
+        with:


Tofel · 2026-03-27T12:28:01Z

will be broken down into more PRs

trigger legacy and CRE tests, when a new CL image is published; trigg…

876150d

…er nightly legacy tests from Docker Build instead of fixed schedule

Tofel added 10 commits March 24, 2026 11:06

add CRE upgrade pipeline

5cdf1fb

bump CTF

1dbb535

pass some buildcmd

84dca00

save Docker logs

7ae5aef

clean up + uncomment df1 compat tests

7994a88

exclude rc and beta refs from upgrade tests

19d8560

support passing custom refs

193f9e1

run CCIP v1.6 tests in build & publish workflow

ef5a5a5

Merge branch 'develop' into dx-3522-trigger-tests-in-release-pipeline

cd014a4

Merge branch 'develop' into dx-3522-trigger-tests-in-release-pipeline

5044762

Tofel marked this pull request as ready for review March 24, 2026 16:14

Tofel requested review from a team as code owners March 24, 2026 16:14

Copilot AI review requested due to automatic review settings March 24, 2026 16:14

product-security-plaid-production Bot requested review from chainchad, ecPablo and ilija42 March 24, 2026 16:15

Copilot started reviewing on behalf of Tofel March 24, 2026 16:15 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Comment thread .github/workflows/system-tests-nightly.yml Outdated

Comment thread .github/workflows/cre-system-tests.yaml Outdated

Comment thread .github/workflows/cre-regression-system-tests.yaml Outdated

Comment thread .github/workflows/build-publish.yml

chainchad reviewed Mar 24, 2026

View reviewed changes

Comment thread .github/workflows/cre-system-tests.yaml

Tofel added 5 commits March 24, 2026 17:23

CR changes

9d3c47c

add reusable action for CL image resolution

fccd023

use shared action for resolving the CL image

8468d33

add action's output

062674d

fix workflows

439f5f1

chainchad previously approved these changes Mar 24, 2026

View reviewed changes

Tofel requested a review from Copilot March 25, 2026 12:44

Copilot started reviewing on behalf of Tofel March 25, 2026 12:45 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

Comment thread .github/workflows/system-tests-nightly.yml Outdated

Comment thread .github/actions/resolve-chainlink-image/action.yaml Outdated

Comment thread .github/actions/resolve-chainlink-image/action.yaml Outdated

Tofel added 5 commits March 25, 2026 14:06

check if env vars are not empty

1d9ff40

update action's reference

899f145

remove PR trigger from nightly tests

af21ea4

allow to run upgrade test on build/publish of rc/beta tags

1752fe7

use a different role for nightly

d35a5d3

Tofel force-pushed the dx-3522-trigger-tests-in-release-pipeline branch 3 times, most recently from 816a8f3 to ba801c1 Compare March 25, 2026 16:40

fix image resolution in nightly

184a528

Tofel force-pushed the dx-3522-trigger-tests-in-release-pipeline branch 2 times, most recently from 0300acb to c1a9994 Compare March 25, 2026 16:55

add login to public ECR to nightly system tests

484efe9

Tofel force-pushed the dx-3522-trigger-tests-in-release-pipeline branch from c1a9994 to 484efe9 Compare March 25, 2026 16:58

add public ECR login to various workflows

f4c4b09

Tofel requested review from chainchad and erikburt March 25, 2026 17:10

Merge branch 'develop' into dx-3522-trigger-tests-in-release-pipeline

ff1fc7f

github-advanced-security AI found potential problems Mar 26, 2026

View reviewed changes

Comment thread .github/workflows/resolve-chainlink-image-tests.yml Fixed

Tofel force-pushed the dx-3522-trigger-tests-in-release-pipeline branch 2 times, most recently from 619e3f0 to a314c58 Compare March 26, 2026 08:30

replace GH action with a bash script, add tests for it and a small wo…

8548103

…rkflow

Tofel force-pushed the dx-3522-trigger-tests-in-release-pipeline branch from a314c58 to 8548103 Compare March 26, 2026 08:31

mchain0 approved these changes Mar 26, 2026

View reviewed changes

erikburt requested changes Mar 26, 2026

View reviewed changes

Tofel closed this Mar 27, 2026

Conversation

Tofel commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cl-sonarqube-production Bot commented Mar 26, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tofel commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Tofel commented Mar 23, 2026 •

edited

Loading

github-actions Bot commented Mar 23, 2026 •

edited

Loading

trunk-io Bot commented Mar 23, 2026 •

edited

Loading