Skip to content

Commit 227bcde

Browse files
derrickawderrickaw
authored
add upgrade gcp bom workflow (#38711)
* initial draft bom workflow * add temp fix * Document new GCP BOM upgrade workflow in README * Fix YAML indentation for push branches trigger * remove tmp push * Rename workflow to Upgrade GCP Libraries BOM * fix gemini comments * move script logic into bomupgrader.py * update workflow logic to only run within 6 days of a release cut
1 parent 27d699e commit 227bcde

4 files changed

Lines changed: 170 additions & 6 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -545,4 +545,5 @@ PostCommit Jobs run in a schedule against master branch and generally do not get
545545
| [ Infrastructure Policy Enforcer ](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_PolicyEnforcer.yml) | N/A | [![.github/workflows/beam_Infrastructure_PolicyEnforcer.yml](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_PolicyEnforcer.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_PolicyEnforcer.yml?query=event%3Aschedule) |
546546
| [ Modify the GCP User Roles according to the infra/users.yml file ](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_UsersPermissions.yml) | N/A | [![.github/workflows/beam_Infrastructure_UsersPermissions.yml](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_UsersPermissions.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_UsersPermissions.yml?query=event%3Aschedule) |
547547
| [ Service Account Keys Management ](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_ServiceAccountKeys.yml) | N/A | [![.github/workflows/beam_Infrastructure_ServiceAccountKeys.yml](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_ServiceAccountKeys.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_ServiceAccountKeys.yml?query=event%3Aschedule) |
548+
| [ Upgrade GCP Libraries BOM ](https://github.com/apache/beam/actions/workflows/beam_Upgrade_GCP_BOM.yml) | N/A | [![.github/workflows/beam_Upgrade_GCP_BOM.yml](https://github.com/apache/beam/actions/workflows/beam_Upgrade_GCP_BOM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Upgrade_GCP_BOM.yml?query=event%3Aschedule) |
548549
| [ Unmanaged Service Accounts Keys Audit ](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_AuditUnmanagedKeys.yml) | N/A | [![.github/workflows/beam_Infrastructure_AuditUnmanagedKeys.yml](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_AuditUnmanagedKeys.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Infrastructure_AuditUnmanagedKeys.yml?query=event%3Aschedule) |

‎.github/workflows/beam_Upgrade_GCP_BOM.yml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: Upgrade GCP Libraries BOM
17+
18+
on:
19+
schedule:
20+
- cron: "0 0 * * 0" # Weekly on Sundays at 00:00 UTC
21+
workflow_dispatch:
22+
23+
permissions:
24+
contents: write
25+
pull-requests: write
26+
checks: read
27+
issues: read
28+
statuses: read
29+
30+
concurrency:
31+
group: '${{ github.workflow }} @ ${{ github.ref }}'
32+
cancel-in-progress: true
33+
34+
jobs:
35+
upgrade_gcp_bom:
36+
runs-on: [self-hosted, ubuntu-24.04, main]
37+
name: Upgrade GCP BOM
38+
steps:
39+
- name: Checkout code
40+
uses: actions/checkout@v6
41+
- name: Setup environment
42+
uses: ./.github/actions/setup-environment-action
43+
with:
44+
python-version: 3.11
45+
java-version: default
46+
go-version: default
47+
- name: Check if new BOM is available
48+
id: check_bom
49+
run: python3 scripts/tools/bomupgrader.py --check
50+
- name: Run bomupgrader
51+
if: steps.check_bom.outputs.should_upgrade == 'true'
52+
run: python3 scripts/tools/bomupgrader.py ${{ steps.check_bom.outputs.latest_version }}
53+
- name: Install gh cli
54+
if: steps.check_bom.outputs.should_upgrade == 'true'
55+
uses: ./.github/actions/setup-gh-cli-linux
56+
- name: Set git config
57+
if: steps.check_bom.outputs.should_upgrade == 'true'
58+
run: |
59+
git config user.name $GITHUB_ACTOR
60+
git config user.email actions@"$RUNNER_NAME".local
61+
- name: Commit Changes and create PR
62+
if: steps.check_bom.outputs.should_upgrade == 'true'
63+
env:
64+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
65+
GH_EVENT: ${{ github.event_name }}
66+
LATEST_VER: ${{ steps.check_bom.outputs.latest_version }}
67+
CURRENT_VER: ${{ steps.check_bom.outputs.current_version }}
68+
run: |
69+
# Take the current date, subtract from a release cut date in the past (June 24, 2026),
70+
# then get the num days % 42 (our release cadence is 42 days).
71+
# This will ensure it only runs the week after a release branch has been cut.
72+
days_diff=$(( ($(date +%s) - $(date --date="260624" +%s) )/(60*60*24)%42 ))
73+
if [[ $GH_EVENT != 'workflow_dispatch' && $days_diff -gt 6 ]]; then
74+
echo "Exiting early. We only update dependencies the week after we cut the release"
75+
exit 0
76+
fi
77+
branchName=upgrade_gcp_bom_${LATEST_VER//./_}
78+
git checkout -b $branchName
79+
git add -A
80+
git diff-index --quiet HEAD || gitdiff=$? || echo $?
81+
if [[ $gitDiff != 0 ]]; then
82+
echo "Changes are ready to commit"
83+
git commit -m "Upgrade GCP Libraries BOM to ${LATEST_VER}" --quiet
84+
git push origin $branchName --quiet
85+
86+
PR_BODY="This PR was created by automation. It upgrades the Google Cloud Platform Libraries BOM from **${CURRENT_VER}** to **${LATEST_VER}** and updates Netty, gRPC, Arrow, Gax, Protobuf, and OpenTelemetry versions to match.
87+
88+
Please review the changes and merge if all tests pass."
89+
90+
GITHUB_PR_URL=$(gh pr create --title "Upgrade GCP Libraries BOM to ${LATEST_VER}" --body "$PR_BODY" --label "dependencies" --base master)
91+
echo "Link of the new PR: $GITHUB_PR_URL"
92+
else
93+
echo "No changes on the files"
94+
fi

‎contributor-docs/release-guide.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1164,10 +1164,16 @@ At the end of the release, go to the GitHub milestones page and mark the recentl
11641164
#### Update the Java BOM
11651165

11661166
Google releases a BOM that pins compatible versions of their Java libraries.
1167-
After the release, try updating the BOM to the latest version.
1167+
After the release, an automated upgrade BOM PR is created or can be triggered manually.
11681168

1169-
To do so, create a draft PR and run test suites following the instructions at
1170-
https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md.
1169+
A PR should have already been created (and possibly merged) by github-actions bot, you should verify that this was done correctly
1170+
by looking at open PRs from that bot - https://github.com/apache/beam/pulls/app%2Fgithub-actions
1171+
1172+
If a PR has not been merged, drive it to completion.
1173+
If no PR was created, triage any failures in https://github.com/apache/beam/actions/workflows/beam_Upgrade_GCP_BOM.yml and manually update the BOM,
1174+
following https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md or simply trigger the workflow again.
1175+
1176+
##### Troubleshooting
11711177

11721178
Triage the test failures and rerun any tests that seem potentially unrelated to the upgrade.
11731179
If there are no test failures due to the BOM upgrade, request review and merge the PR as normal.

‎scripts/tools/bomupgrader.py‎

Lines changed: 66 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import re
2121
import subprocess
2222
import sys
23+
import urllib.request
2324
"""
2425
This Python script is used for upgrading the GCP-BOM in BeamModulePlugin.
2526
Specifically, it
@@ -43,6 +44,59 @@
4344
# To format: yapf --style sdks/python/setup.cfg --in-place scripts/tools/bomupgrader.py
4445

4546

47+
def get_latest_bom():
48+
url = "https://repo1.maven.org/maven2/com/google/cloud/libraries-bom/maven-metadata.xml"
49+
with urllib.request.urlopen(url, timeout=15) as response:
50+
xml = response.read().decode('utf-8')
51+
match = re.search(r'<release>([^<]+)</release>', xml)
52+
if match:
53+
return match.group(1)
54+
raise RuntimeError("Could not find latest release in Maven metadata")
55+
56+
57+
def get_current_bom():
58+
path = "buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy"
59+
with open(path) as f:
60+
content = f.read()
61+
match = re.search(
62+
r'google_cloud_platform_libraries_bom\s*:\s*[\"\']com\.google\.cloud:libraries-bom:([0-9.]+)[\"\']',
63+
content)
64+
if match:
65+
return match.group(1)
66+
raise RuntimeError(
67+
"Could not find current libraries-bom in BeamModulePlugin.groovy")
68+
69+
70+
def to_tuple(version_str):
71+
parts = []
72+
for part in version_str.split('.'):
73+
match = re.match(r'^(\d+)', part)
74+
parts.append(int(match.group(1)) if match else 0)
75+
return tuple(parts)
76+
77+
78+
def check_bom():
79+
latest = get_latest_bom()
80+
current = get_current_bom()
81+
print(f"Latest libraries-bom version: {latest}")
82+
print(f"Current libraries-bom version: {current}")
83+
84+
should_upgrade = to_tuple(latest) > to_tuple(current)
85+
86+
github_output = os.getenv('GITHUB_OUTPUT')
87+
if github_output:
88+
with open(github_output, 'a') as f:
89+
f.write(f"should_upgrade={str(should_upgrade).lower()}\n")
90+
f.write(f"latest_version={latest}\n")
91+
f.write(f"current_version={current}\n")
92+
93+
if should_upgrade:
94+
print("A newer version of libraries-bom is available. Upgrade needed.")
95+
else:
96+
print("libraries-bom is up-to-date.")
97+
return should_upgrade, latest
98+
99+
46100
class BeamModulePluginProcessor:
47101
# Known dependencies managed by GCP BOM and also used by Beam.
48102
# We only need to have one dependency for each project to figure out the target version
@@ -313,7 +367,16 @@ def run(self):
313367
if __name__ == '__main__':
314368
logging.getLogger().setLevel(logging.INFO)
315369
if len(sys.argv) < 2:
316-
print("Usage: python scripts/tools/bomupgrader.py target_version")
370+
print("Usage: python scripts/tools/bomupgrader.py [--check | latest | target_version]")
317371
exit(1)
318-
processor = BeamModulePluginProcessor(sys.argv[1])
319-
processor.run()
372+
373+
arg = sys.argv[1]
374+
if arg in ['--check', '--check-only']:
375+
check_bom()
376+
else:
377+
if arg in ['latest', '--latest']:
378+
_, target_ver = check_bom()
379+
else:
380+
target_ver = arg
381+
processor = BeamModulePluginProcessor(target_ver)
382+
processor.run()

0 commit comments

Comments
 (0)