Skip to content

Commit 47a2834

Browse files
authored
ci(showcase): replace hermetic build golden tests with Librarian (googleapis#13569)
Replace the hermetic build code generation and verification workflow for the showcase client with direct Librarian execution. - Update `generate_showcase.sh` to run `librarian generate showcase` directly, with the Librarian version defined in the workspace's `librarian.yaml` configuration. - Simplify `verify.sh` to generate the code in-place and use `git status` on the generated source directories to check for drift, eliminating the complex temporary directory and symlink mirroring setups. - Add Go, Python, protoc, and Librarian setup steps to the showcase workflow runner in `showcase.yaml` so tests can execute librarian generate. - Update `java-showcase/README.md` to document Go, protoc, and Python virtual environment setup instructions. Choose to put environment setups outside the script to prevent it from intruding on developers' local configurations, keep execution fast. Note that there are on-going work to simplify these setups (googleapis/librarian#6558 to make protoc part of install; https://github.com/googleapis/librarian/milestone/139 to migrate away from python owlbot.py) Fixes googleapis/librarian#6393
1 parent 16df225 commit 47a2834

5 files changed

Lines changed: 81 additions & 108 deletions

File tree

.github/workflows/showcase.yaml

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,13 +115,46 @@ jobs:
115115
env:
116116
BUILD_SUBDIR: sdk-platform-java
117117
JOB_TYPE: install
118+
- uses: actions/setup-go@v5
119+
with:
120+
go-version: 'stable'
121+
- name: Install protoc
122+
run: |
123+
set -e
124+
VERSION="33.2"
125+
curl -fsSL --retry 5 --retry-delay 15 -o /tmp/protoc.zip "https://github.com/protocolbuffers/protobuf/releases/download/v$VERSION/protoc-$VERSION-linux-x86_64.zip"
126+
cd /usr/local
127+
sudo unzip -o /tmp/protoc.zip
128+
protoc --version
129+
- uses: actions/setup-python@v5
130+
with:
131+
python-version: "3.12"
132+
cache: 'pip'
133+
- name: Install Librarian and tools
134+
env:
135+
PYTHONPATH: ${{ github.workspace }}/sdk-platform-java/hermetic_build/library_generation/owlbot
136+
run: |
137+
V=$(go run github.com/googleapis/librarian/cmd/librarian@latest config get version)
138+
echo "Installing librarian version $V"
139+
go install github.com/googleapis/librarian/cmd/librarian@$V
140+
librarian install
141+
- name: Setup Java 17 for Golden Tests
142+
uses: actions/setup-java@v4
143+
with:
144+
java-version: 17
145+
distribution: temurin
118146
- name: Showcase golden tests
119147
working-directory: java-showcase
120148
run: |
121149
mvn test \
122150
-P enable-golden-tests \
123151
--batch-mode \
124152
--no-transfer-progress
153+
- name: Restore Matrix Java
154+
uses: actions/setup-java@v4
155+
with:
156+
java-version: ${{ matrix.java }}
157+
distribution: temurin
125158
- name: Parse showcase version
126159
working-directory: java-showcase/gapic-showcase
127160
run: echo "SHOWCASE_VERSION=$(mvn help:evaluate -Dexpression=gapic-showcase.version -q -DforceStdout)" >> "$GITHUB_ENV"

java-showcase/README.md

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ the Showcase API which can communicate with a local Showcase server to perform i
77

88
## Requirements
99

10-
* Install [Go](https://go.dev) in your `PATH`.
10+
* Install [Go](https://go.dev) in your `PATH` (version 1.20 or later).
11+
* Install the [protoc](https://github.com/protocolbuffers/protobuf/releases) compiler in your `PATH`.
1112

1213
## Installing the Server
1314

@@ -73,19 +74,40 @@ mvn verify -P enable-integration-tests
7374

7475
## Running the Golden tests
7576

76-
**NOTE** This requires Docker to be installed in your machine.
77-
Open a new terminal window in the root project directory.
77+
This verify check compares the current checked-in files in `java-showcase` against a newly generated client.
78+
79+
### Prerequisites (One-time Setup)
80+
Make sure you have **Go** and **protoc** installed and available in your `PATH` (see the [Requirements](#requirements) section), then run the following setup steps from the repository root:
81+
82+
```shell
83+
# Setup and activate Python virtual environment
84+
python -m venv .venv
85+
source .venv/bin/activate
86+
87+
# Configure pip for external sources
88+
pip config --venv set global.extra-index-url https://pypi.org/simple
89+
90+
# Install and configure Java tools (takes around 5 mins)
91+
V=$(go run github.com/googleapis/librarian/cmd/librarian@latest config get version)
92+
go run github.com/googleapis/librarian/cmd/librarian@${V} install
93+
```
94+
95+
### Running the tests
96+
Once the prerequisites are installed and the virtual environment is activated, run:
7897

7998
```shell
8099
cd java-showcase
81100
mvn verify -P enable-golden-tests
82101
```
83102

103+
> [!IMPORTANT]
104+
> The golden tests run the generator and compare the entire `java-showcase` directory against the newly generated state using `git diff`. For these tests to pass, you must have a completely clean git tree within the `java-showcase` directory before running them. Any uncommitted changes or untracked files will cause the tests to fail.
84105
85106
## Update the Golden Showcase Files
86107

87-
**NOTE** This requires Docker to be installed in your machine.
88-
Open a new terminal window in the root project directory.
108+
To regenerate the showcase library files and update them in place:
109+
110+
Make sure the Python virtual environment is activated, then run:
89111

90112
```shell
91113
# In repository's root directory

java-showcase/pom.xml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -251,8 +251,6 @@
251251
<executable>bash</executable>
252252
<arguments>
253253
<argument>scripts/generate_showcase.sh</argument>
254-
<argument>--replace</argument>
255-
<argument>true</argument>
256254
</arguments>
257255
</configuration>
258256
</execution>
Lines changed: 10 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -1,96 +1,19 @@
11
#!/bin/bash
2-
# Generates the showcase library using the docker image, which is built
3-
# from the current state of the repo in order to test local changes.
2+
# Generates the showcase library using librarian.
43
set -ex
54

65
echo "******** Generating Showcase ********"
76

8-
trap cleanup ERR
9-
107
readonly ROOT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )/../.."
11-
pushd "${ROOT_DIR}"
12-
source "${ROOT_DIR}/java-showcase/scripts/showcase_utilities.sh"
13-
14-
cleanup() {
15-
if [[ -z "${api_def_dir}" ]]; then
16-
rm -rf "${api_def_dir}"
17-
fi
18-
if [[ -z "${showcase_def_dir}" ]]; then
19-
rm -rf "${showcase_def_dir}"
20-
fi
21-
}
22-
23-
while [[ $# -gt 0 ]]; do
24-
key="$1"
25-
case "${key}" in
26-
--replace)
27-
replace="$2"
28-
shift
29-
;;
30-
*)
31-
echo "Invalid option: [$1]"
32-
exit 1
33-
;;
34-
esac
35-
shift
36-
done
37-
38-
if [ -z "${replace}" ]; then
39-
replace="false"
40-
fi
41-
42-
43-
# download api definitions from googleapis repository
44-
googleapis_commitish=$(grep googleapis_commitish generation_config.yaml | cut -d ":" -f 2 | xargs)
45-
api_def_dir=$(mktemp -d)
46-
git clone https://github.com/googleapis/googleapis.git "${api_def_dir}"
47-
48-
pushd "${api_def_dir}"
49-
git checkout "${googleapis_commitish}"
50-
# for local setups, we avoid permission issues when the docker image
51-
# performs version-dependent operations.
52-
rm -rf ".git/"
53-
popd
54-
55-
append_showcase_to_api_defs "${api_def_dir}"
56-
57-
if [[ -f "image-id" ]]; then
58-
echo "image already exists:"
59-
cat image-id
60-
else
61-
echo "building docker image"
62-
DOCKER_BUILDKIT=1 docker build --file sdk-platform-java/.cloudbuild/library_generation/library_generation.Dockerfile --iidfile image-id .
63-
fi
64-
65-
if [[ "${replace}" == "true" ]]; then
66-
generated_files_dir="${ROOT_DIR}"
67-
else
68-
export generated_files_dir=$(mktemp -d)
69-
# here we store the generated library location for upstream scripts to use
70-
# it.
71-
echo "${generated_files_dir}/java-showcase" > "${ROOT_DIR}/generated-showcase-location"
72-
# we prepare the temp folder with the minimal setup to perform an incremental
73-
# generation.
74-
pushd "${ROOT_DIR}"
75-
cp -r generation_config.yaml java-showcase/ versions.txt "${generated_files_dir}"
76-
popd #ROOT_DIR
77-
fi
8+
# Export PYTHONPATH so that owlbot.py scripts run by Librarian
9+
# can resolve local synthtool under the hermetic build directory.
10+
export PYTHONPATH="${ROOT_DIR}/sdk-platform-java/hermetic_build/library_generation/owlbot"
11+
cd "${ROOT_DIR}"
7812

79-
pushd sdk-platform-java
80-
GENERATOR_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout -pl gapic-generator-java)
81-
popd
13+
readonly LIBRARIAN_VER=$(go run github.com/googleapis/librarian/cmd/librarian@latest config get version)
14+
readonly LIBRARIAN_CMD="go run github.com/googleapis/librarian/cmd/librarian@${LIBRARIAN_VER}"
8215

83-
echo "generating showcase"
84-
workspace_name="/workspace"
85-
docker run \
86-
--rm \
87-
-u "$(id -u):$(id -g)" \
88-
-v "${generated_files_dir}:${workspace_name}" \
89-
-v "${api_def_dir}:${workspace_name}/googleapis" \
90-
-e GENERATOR_VERSION="${GENERATOR_VERSION}" \
91-
"$(cat image-id)" \
92-
--generation-config-path="${workspace_name}/generation_config.yaml" \
93-
--library-names="showcase" \
94-
--api-definitions-path="${workspace_name}/googleapis"
16+
echo "generating showcase in place"
17+
$LIBRARIAN_CMD generate showcase
9518

96-
echo "generated showcase library in ${generated_files_dir}"
19+
echo "generated showcase library"

java-showcase/scripts/verify.sh

Lines changed: 11 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,18 @@
1-
#!/bin/sh
2-
# This script generates showcase in a temporary/untracked folder and compares
3-
# its contents with the actual showcase libraries.
1+
#!/bin/bash
2+
# This script generates showcase in place and checks for git diffs to verify.
43

54
echo "******** Verifying Showcase ********"
65

7-
set -oxe
6+
set -ex
87
readonly SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
98
readonly ROOT_DIR="${SCRIPT_DIR}/../.."
10-
source "${SCRIPT_DIR}/showcase_utilities.sh"
119
readonly SHOWCASE_DIR="${SCRIPT_DIR}/.."
10+
# generate sources in place
11+
bash "${SCRIPT_DIR}/generate_showcase.sh"
1212

13-
# generate sources
14-
bash "${SCRIPT_DIR}/generate_showcase.sh" --replace "false"
15-
16-
generated_library_location=$(cat "${ROOT_DIR}/generated-showcase-location")
17-
18-
# compare library
19-
diff -ru "${SHOWCASE_DIR}/" "${generated_library_location}"
20-
21-
cleanup $SCRIPT_DIR
13+
# check if there are changes in java-showcase directory
14+
if [ -n "$(git status --porcelain ${SHOWCASE_DIR})" ]; then
15+
git diff ${SHOWCASE_DIR}
16+
echo "Error: Showcase generated files are out of sync. Please run 'mvn compile -P update' inside java-showcase to update them."
17+
exit 1
18+
fi

0 commit comments

Comments
 (0)