Skip to content

Commit bf53ea3

Browse files
dlvenabledlvenable
authored
Updates the release process to run automatically and fully with new OpenSearch project rules. (#6921)
Updates the release process to run automatically and fully with new OpenSearch project rules. The OpenSearch project no longer supports tagging in workflows or use of app tokens. The release GitHub Action currently creates the tag for a release. Now that this is unsupported, the release GHA validates that the tag exists and matches what we expect from the gradle.properties file. Both the release and release-prepare-branch GitHub Actions currently create PRs against the repo. The new approach uses a dedicated fork for creating these PRs. Access to the fork is granted via a personal access token. This commit also updates related documentation for setting up a repository for releases. Resolves #6912. Signed-off-by: David Venable <dlv@amazon.com>
1 parent f165af3 commit bf53ea3

3 files changed

Lines changed: 212 additions & 31 deletions

File tree

‎.github/workflows/release-prepare-branch.yml‎

Lines changed: 53 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,33 @@
1010
# This workflow automates the preparation of patch release branches.
1111
# It validates the release branch, updates gradle.properties to the next version
1212
# generates the THIRD-PARTY file, and creates a pull request with all changes.
13+
#
14+
# The OpenSearch project does not permit this repository's workflows to push
15+
# branches and open pull requests against itself. Instead, the pull request
16+
# branch is pushed to a fork and the PR is opened from that fork against this
17+
# repository. This requires two settings to be configured:
18+
#
19+
# vars.RELEASE_FORK_REPOSITORY - the fork the PR branch is pushed to, as
20+
# 'owner/repo'. It must be a real GitHub fork
21+
# of this repository (same fork network), as
22+
# cross-repository PRs are only allowed within
23+
# a fork network.
24+
# secrets.OPENSEARCH_CI_BOT_TOKEN - a classic personal access token with the
25+
# 'public_repo' (or 'repo' for private
26+
# repositories) and 'workflow' scopes. The
27+
# action uses it to push the branch to the
28+
# fork and to open the PR against this
29+
# repository, so its owner must have write
30+
# access to the fork. The 'workflow' scope is
31+
# required because the pushed branch may modify
32+
# files under .github/workflows; GitHub rejects
33+
# such pushes from a token without it. A
34+
# fine-grained token does not work here because
35+
# the single token must act on two repositories.
36+
#
37+
# The PR base is always this repository (the one running the workflow). To test
38+
# on your own fork, run this workflow from your fork and point
39+
# RELEASE_FORK_REPOSITORY at a fork of your fork within the same network.
1340

1441
name: Prepare Release Branch
1542

@@ -27,6 +54,12 @@ jobs:
2754
steps:
2855
- name: Checkout Data Prepper
2956
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
57+
with:
58+
# Do not persist the checkout credentials in the local git config.
59+
# When pushing to a fork, peter-evans/create-pull-request adds its own
60+
# authentication, and a persisted Authorization header would collide
61+
# with it, causing 'Duplicate header: "Authorization"' (HTTP 400).
62+
persist-credentials: false
3063

3164
- name: Validate release branch
3265
id: validate_branch
@@ -50,6 +83,20 @@ jobs:
5083
echo "✅ Valid release branch: $BRANCH_NAME"
5184
echo "branch=$BRANCH_NAME" >> $GITHUB_OUTPUT
5285
86+
- name: Validate fork configuration
87+
run: |
88+
if [ -z "${{ vars.RELEASE_FORK_REPOSITORY }}" ]; then
89+
echo "::error::The RELEASE_FORK_REPOSITORY variable is not set."
90+
echo "Set it to the 'owner/repo' of a fork of this repository that the release PR branch will be pushed to."
91+
exit 1
92+
fi
93+
if [ -z "${{ secrets.OPENSEARCH_CI_BOT_TOKEN }}" ]; then
94+
echo "::error::The OPENSEARCH_CI_BOT_TOKEN secret is not set."
95+
echo "Set it to a token with write access to ${{ vars.RELEASE_FORK_REPOSITORY }} and permission to create pull requests against this repository."
96+
exit 1
97+
fi
98+
echo "Pull request branch will be pushed to fork: ${{ vars.RELEASE_FORK_REPOSITORY }}"
99+
53100
- name: Set up JDK
54101
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
55102
with:
@@ -91,18 +138,12 @@ jobs:
91138
92139
echo "Successfully generated THIRD-PARTY file"
93140
94-
- name: GitHub App token
95-
id: github_app_token
96-
uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2
97-
with:
98-
app_id: ${{ secrets.APP_ID }}
99-
private_key: ${{ secrets.APP_PRIVATE_KEY }}
100-
101141
- name: Create Pull Request
102142
id: create_pr
103143
uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6
104144
with:
105-
token: ${{ steps.github_app_token.outputs.token }}
145+
token: ${{ secrets.OPENSEARCH_CI_BOT_TOKEN }}
146+
push-to-fork: ${{ vars.RELEASE_FORK_REPOSITORY }}
106147
add-paths: |
107148
gradle.properties
108149
THIRD-PARTY
@@ -129,6 +170,7 @@ jobs:
129170
130171
After merging this PR:
131172
- [ ] Prepare release notes (see [release notes script](release/script/release-notes/README.md))
173+
- [ ] Open an issue in the [opensearch-project/.github](https://github.com/opensearch-project/.github) repository requesting the release tag `${{ steps.version.outputs.version }}` be created and pushed.
132174
- [ ] Run the [Release Artifacts workflow](https://github.com/opensearch-project/data-prepper/actions/workflows/release.yml)
133175
- [ ] Approve the release issue (requires 2 maintainer approvals)
134176
- [ ] Update the draft release with release notes
@@ -153,8 +195,9 @@ jobs:
153195
run: |
154196
echo "::error::Failed to create pull request"
155197
echo "This could be due to:"
156-
echo " - Missing or invalid GitHub App credentials"
157-
echo " - Insufficient permissions"
198+
echo " - An invalid or expired OPENSEARCH_CI_BOT_TOKEN"
199+
echo " - The token lacking write access to ${{ vars.RELEASE_FORK_REPOSITORY }} or permission to create pull requests"
200+
echo " - RELEASE_FORK_REPOSITORY (${{ vars.RELEASE_FORK_REPOSITORY }}) not being a fork of this repository in the same fork network"
158201
echo " - Network issues"
159202
echo "Please check the logs above for more details"
160203
exit 1

‎.github/workflows/release.yml‎

Lines changed: 80 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,32 @@
77
# compatible open source license.
88
#
99

10+
# This workflow builds and validates the release artifacts, drafts the GitHub
11+
# release, and opens the changelog pull request.
12+
#
13+
# The OpenSearch project does not permit this repository's workflows to push
14+
# branches and open pull requests against itself. Instead, the changelog PR
15+
# branch is pushed to a fork and the PR is opened from that fork against this
16+
# repository. This requires two settings to be configured:
17+
#
18+
# vars.RELEASE_FORK_REPOSITORY - the fork the PR branch is pushed to, as
19+
# 'owner/repo'. It must be a real GitHub fork
20+
# of this repository (same fork network), as
21+
# cross-repository PRs are only allowed within
22+
# a fork network.
23+
# secrets.OPENSEARCH_CI_BOT_TOKEN - a classic personal access token with the
24+
# 'public_repo' (or 'repo' for private
25+
# repositories) and 'workflow' scopes. The
26+
# action uses it to push the branch to the
27+
# fork and to open the PR against this
28+
# repository, so its owner must have write
29+
# access to the fork. The 'workflow' scope is
30+
# required because the pushed branch may modify
31+
# files under .github/workflows; GitHub rejects
32+
# such pushes from a token without it. A
33+
# fine-grained token does not work here because
34+
# the single token must act on two repositories.
35+
1036
name: Release Artifacts
1137

1238
on:
@@ -30,9 +56,60 @@ permissions:
3056
pull-requests: write
3157

3258
jobs:
59+
# Verifies the release preconditions
60+
# 1. The release tag must already exist. A maintainer requests the tag via an issue in the
61+
# opensearch-project/.github repository before running this workflow.
62+
# 2. The fork settings used to open the changelog pull request must be set.
63+
verify-preconditions:
64+
runs-on: ubuntu-latest
65+
timeout-minutes: 5
66+
67+
steps:
68+
- name: Checkout Data Prepper
69+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
70+
- name: Get Version
71+
run: grep '^version=' gradle.properties >> $GITHUB_ENV
72+
- name: Verify release tag exists
73+
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
74+
with:
75+
github-token: ${{ github.TOKEN }}
76+
script: |
77+
const tag = '${{ env.version }}';
78+
try {
79+
await github.rest.git.getRef({
80+
owner: context.repo.owner,
81+
repo: context.repo.repo,
82+
ref: `tags/${tag}`
83+
});
84+
core.info(`Found tag ${tag}.`);
85+
} catch (error) {
86+
if (error.status === 404) {
87+
core.setFailed(
88+
`Release tag '${tag}' does not exist. Request the tag by opening an issue in the ` +
89+
`opensearch-project/.github repository before running this workflow.`
90+
);
91+
} else {
92+
throw error;
93+
}
94+
}
95+
- name: Validate fork configuration
96+
run: |
97+
if [ -z "${{ vars.RELEASE_FORK_REPOSITORY }}" ]; then
98+
echo "::error::The RELEASE_FORK_REPOSITORY variable is not set."
99+
echo "Set it to the 'owner/repo' of a fork of this repository that the changelog PR branch will be pushed to."
100+
exit 1
101+
fi
102+
if [ -z "${{ secrets.OPENSEARCH_CI_BOT_TOKEN }}" ]; then
103+
echo "::error::The OPENSEARCH_CI_BOT_TOKEN secret is not set."
104+
echo "Set it to a token with write access to ${{ vars.RELEASE_FORK_REPOSITORY }} and permission to create pull requests against this repository."
105+
exit 1
106+
fi
107+
echo "Changelog pull request branch will be pushed to fork: ${{ vars.RELEASE_FORK_REPOSITORY }}"
108+
33109
build:
34110
runs-on: ubuntu-latest
35111
timeout-minutes: 50
112+
needs: verify-preconditions
36113

37114
steps:
38115
- name: Set up JDK
@@ -181,24 +258,12 @@ jobs:
181258
echo 'release_major_tag: ${{ github.event.inputs.release-major-tag }}' >> release-description.yaml
182259
echo 'release_latest_tag: ${{ github.event.inputs.release-latest-tag }}' >> release-description.yaml
183260
184-
- name: Create tag
185-
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
186-
with:
187-
github-token: ${{ github.TOKEN }}
188-
script: |
189-
github.rest.git.createRef({
190-
owner: context.repo.owner,
191-
repo: context.repo.repo,
192-
ref: 'refs/tags/${{ env.version }}',
193-
sha: context.sha
194-
})
195-
196261
- name: Draft release
197262
uses: softprops/action-gh-release@de2c0eb89ae2a093876385947365aca7b0e5f844 # v1
198263
with:
199264
draft: true
200265
name: '${{ env.version }}'
201-
tag_name: 'refs/tags/${{ env.version }}'
266+
tag_name: '${{ env.version }}'
202267
files: |
203268
release-description.yaml
204269
@@ -227,17 +292,11 @@ jobs:
227292
- name: Generate Changelog
228293
run: release/script/changelog/generate-changelog.sh ${{ needs.promote.outputs.version }}
229294

230-
- name: GitHub App token
231-
id: github_app_token
232-
uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2
233-
with:
234-
app_id: ${{ secrets.APP_ID }}
235-
private_key: ${{ secrets.APP_PRIVATE_KEY }}
236-
237295
- name: Create Pull Request
238296
uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6
239297
with:
240-
token: ${{ steps.github_app_token.outputs.token }}
298+
token: ${{ secrets.OPENSEARCH_CI_BOT_TOKEN }}
299+
push-to-fork: ${{ vars.RELEASE_FORK_REPOSITORY }}
241300
add-paths: |
242301
release/release-notes/data-prepper.change-log-${{ needs.promote.outputs.version }}.md
243302
commit-message: 'Add Data Prepper ${{ needs.promote.outputs.version }} changelog.'

‎release/README.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,82 @@ It is not a project and instead holds release notes and change logs for Data Pre
99
## Performing a Release
1010

1111
See [RELEASING.md](../RELEASING.md) for instructions to follow for any release.
12+
13+
## Configuring the GitHub repo
14+
15+
Running a release requires both AWS staging resources and a number of GitHub Actions secrets and variables.
16+
The release workflows ([Prepare Release Branch](../.github/workflows/release-prepare-branch.yml) and
17+
[Release Artifacts](../.github/workflows/release.yml)) read these to build and upload the artifacts and to
18+
open their pull requests (PRs).
19+
20+
### Deploy the AWS staging resources
21+
22+
The release build uploads archives and Maven artifacts to AWS S3 and publishes Docker images to a staging ECR
23+
repository. These resources, and the IAM role that GitHub Actions assumes to access them, are provisioned by
24+
the [staging-resources-cdk](staging-resources-cdk/README.md) project. Follow that project's README to install
25+
the CDK and deploy the stacks before running a release.
26+
27+
The CDK deployment provides the values you will use for the `RELEASE_IAM_ROLE`, `ARCHIVES_BUCKET_NAME`,
28+
`ARCHIVES_PUBLIC_URL`, and `ECR_REPOSITORY_URL` secrets described below.
29+
30+
### Configure a fork for pull requests
31+
32+
The OpenSearch project does not permit this repository's workflows to push branches and open PRs
33+
against itself. Instead, the release workflows push their PR branches to a fork and open the PR from that fork
34+
back against this repository. This applies to both the release preparation PR and the changelog PR.
35+
36+
To run a release build, you must have a fork of Data Prepper used for staging these changes and PRs.
37+
This must be an actual GitHub fork of the repository you are releasing, because GitHub only permits
38+
cross-repository PRs between repositories in the same fork network.
39+
40+
The fork is identified by the `RELEASE_FORK_REPOSITORY` variable, and the token used to push to it and open
41+
the PR is the `OPENSEARCH_CI_BOT_TOKEN` secret. `OPENSEARCH_CI_BOT_TOKEN` must be a **classic** personal access token
42+
(PAT) meeting the following requirements.
43+
44+
* It is a classic PAT. A fine-grained token does not work because the single token must act on two
45+
repositories (the fork it pushes to and this repository it opens the PR against).
46+
* Its owner has write access to the fork named in `RELEASE_FORK_REPOSITORY`.
47+
* It has the `public_repo` scope (or `repo` if the repository is private).
48+
* It has the `workflow` scope. This is required because the pushed branch may include changes to files
49+
under `.github/workflows`, and GitHub rejects such pushes from a token without it.
50+
51+
Because a classic PAT acts as its owner across all of their repositories, use a dedicated release or bot
52+
account to own this token rather than a maintainer's personal account, and set a short expiration so the
53+
token is rotated regularly.
54+
55+
### Secrets and variables
56+
57+
Configure the following on the repository that runs the release workflows. Create variables as
58+
[repository variables](https://docs.github.com/en/actions/learn-github-actions/variables) and secrets as
59+
[repository secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
60+
61+
| Name | Type | Source | Description |
62+
| ---- | ---- | ------ | ----------- |
63+
| `RELEASE_FORK_REPOSITORY` | Variable | You | The `owner/repo` of the fork the PR branches are pushed to, for example `opensearch-ci-bot/data-prepper`. |
64+
| `OPENSEARCH_CI_BOT_TOKEN` | Secret | You | A classic PAT used to push the PR branch to the fork and open the PR. See the requirements above. |
65+
| `RELEASE_IAM_ROLE` | Secret | CDK | The ARN of the IAM role GitHub Actions assumes to access the staging resources. |
66+
| `ARCHIVES_BUCKET_NAME` | Secret | CDK | The name of the S3 bucket the release archives and Maven artifacts are uploaded to. |
67+
| `ARCHIVES_PUBLIC_URL` | Secret | CDK | The public base URL the uploaded archives are served from, used by the tarball smoke tests. |
68+
| `ECR_REPOSITORY_URL` | Secret | CDK | The URL of the staging ECR repository the Docker images are pushed to. |
69+
70+
## Testing release changes
71+
72+
You can test release changes by running the release workflows from your own fork of Data Prepper.
73+
The release workflows always open their PRs against the repository that runs them, so when you run them
74+
from your fork, the PRs target your fork.
75+
76+
To set this up:
77+
* Run the workflows from your fork of Data Prepper (for example `your-user/data-prepper`).
78+
* Create a second fork of your fork in an account or organization you control (for example
79+
`your-test-org/data-prepper`). This second fork is the head repository that the PR branches are pushed to.
80+
It is required because GitHub does not allow a PR where the head and base are the same repository.
81+
Note that GitHub permits only one fork per account, so the second fork must live in a different account
82+
or organization; both forks remain in the same fork network, which is what allows the cross-repository PR.
83+
* Set `RELEASE_FORK_REPOSITORY` and `OPENSEARCH_CI_BOT_TOKEN` on your fork as described in
84+
[Configuring the GitHub repo](#configuring-the-github-repo), pointing `RELEASE_FORK_REPOSITORY` at the
85+
second fork.
86+
87+
To test, you must also make a few manual modifications that you will not check in.
88+
* Update the `get_approvers` step with `minimum-approvals: 1`. Otherwise, you will be blocked in the promote step.
89+
* Change your `CODEOWNERS` file to have only your user. Without this, the promote step will try to assign this to users who are not in your fork and then fail.
90+
* You should consider updating the "Build Jar Files" step to only `assemble` instead of `build` to speed up your tests. `./gradlew --parallel --max-workers 2 assemble`

0 commit comments

Comments
 (0)