Skip to content

Commit 341779c

Browse files
diegomarquezpdiegomarquezp
authored
chore: adding a GitHub Action for generating new clients using the hermetic build scripts (#10488)
* chore: add new-client for hermetic build * enable yaml insertion * normalize yaml format * add requirements file * add gh action * add pull_request event for testing * add requirements.txt * use requirements with hashes * add requirement hashes * fix requirements.in * remove testing event * fix requirement hashes * fix generation script call * remove usage of googleapis-gen-url * fix gapic entry generation * add debug print statement * remove mistakenly added chat library * suppress debug output * fix pr message, fix pr label * fix pr description message * improve pr descriptoin * fix comment * add readme * add advanced options * fix syntax * clarify type of workflow * fix advanced options * checkout latest fix in main branch * remove transport form workflow options * change destination_name to library_name, remove cloud_api * remove library existence check * change product_docs to product_documentation * Revert "change product_docs to product_documentation" This reverts commit d4a4909. * add api_reference * add codeowner_team * add excluded_dependencies * add excluded_poms * add googleapis_commitish * add mutually exclusive logic for group_id and distribution_name * add issue_tracker * add extra_versioned_modules * remove untracked file * improve logic for inferring yaml variables * remove debug file * compute all proto_path verions * update googleapis_commitish for testing * add cleanup of googleapis folder * add xtrace * update dependencies * update dependencies ii * remove owlbot label * add owlbot run label * fix library_name * restore label * reorganize required params * check if library is releasable * sync generation_config.yaml * temporarily use latest commitish * Revert "temporarily use latest commitish" This reverts commit fe525b5. * add python script to parse arguments * fix argument generation * fix api-shortname parameter * restart tests * fix documentation and comments * add `library_name` explanation * clarify api_shortname * rename to add-new-client-config * remove redundant library_name assignment * support for versioned proto_paths only * instructions for multiple proto_paths * restore config yaml * restore generation config
1 parent 9e38504 commit 341779c

File tree

6 files changed

+717
-0
lines changed

6 files changed

+717
-0
lines changed

‎.github/workflows/generate_new_client_hermetic_build.yaml‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
name: Generate new GAPIC client library (Hermetic Build)
2+
on:
3+
workflow_dispatch:
4+
# some inputs are ommited due to limit of 10 input arguments
5+
inputs:
6+
api_shortname:
7+
required: true
8+
type: string
9+
description: "`api_shortname`: Name for the new directory name and (default) artifact name"
10+
name_pretty:
11+
required: true
12+
type: string
13+
description: "`name_pretty`: The human-friendly name that appears in README.md"
14+
api_description:
15+
required: true
16+
description: "`api_description`: Description that appears in README.md"
17+
proto_path:
18+
required: true
19+
type: string
20+
description: |
21+
`proto_path`: Path to proto file from the root of the googleapis repository to the
22+
directory that contains the proto files (with the version).
23+
For example, to generate `v2` of google/cloud/library,
24+
you must pass google/cloud/library/v2
25+
product_docs:
26+
required: true
27+
type: string
28+
description: "`product_docs`: Documentation URL that appears in README.md"
29+
rest_docs:
30+
required: false
31+
type: string
32+
description: |
33+
`rest_docs`: If it exists, link to the REST Documentation for a service
34+
rpc_docs:
35+
required: false
36+
type: string
37+
description: |
38+
`rpc_docs`: If it exists, link to the RPC Documentation for a service
39+
library_name:
40+
required: false
41+
type: string
42+
description: |
43+
`library_name`: The directory name of the new library. By default it's
44+
java-<api_shortname>
45+
distribution_name:
46+
required: false
47+
type: string
48+
description: |
49+
`distribution_name`: Maven coordinates of the generated library. By default it's
50+
com.google.cloud:google-cloud-<api_shortname>
51+
52+
jobs:
53+
generate:
54+
runs-on: ubuntu-22.04
55+
steps:
56+
- uses: actions/checkout@v3
57+
- uses: actions/setup-python@v4
58+
with:
59+
python-version: '3.9'
60+
cache: 'pip' # caching pip dependencies
61+
- name: Install add-new-client-config.py dependencies
62+
run: pip install --require-hashes -r generation/new_client_hermetic_build/requirements.txt
63+
- name: Add entry to generation_config.yaml
64+
id: config_generation
65+
run: |
66+
set -x
67+
arguments=$(python generation/new_client_hermetic_build/generate-arguments.py)
68+
echo "::set-output name=new_library_args::${arguments}"
69+
echo "${arguments}" \
70+
| xargs python generation/new_client_hermetic_build/add-new-client-config.py add-new-library
71+
env:
72+
API_SHORTNAME: ${{ github.event.inputs.api_shortname }}
73+
NAME_PRETTY: ${{ github.event.inputs.name_pretty }}
74+
PROTO_PATH: ${{ github.event.inputs.proto_path }}
75+
PRODUCT_DOCS: ${{ github.event.inputs.product_docs }}
76+
REST_DOCS: ${{ github.event.inputs.rest_docs }}
77+
RPC_DOCS: ${{ github.event.inputs.rpc_docs }}
78+
API_DESCRIPTION: ${{ github.event.inputs.api_description }}
79+
LIBRARY_NAME: ${{ github.event.inputs.library_name }}
80+
DISTRIBUTION_NAME: ${{ github.event.inputs.distribution_name }}
81+
- name: setup docker environment
82+
shell: bash
83+
run: |
84+
set -x
85+
# we create a volume pointing to `pwd` (google-cloud-java) that will
86+
# be referenced by the container and its children
87+
if [[ $(docker volume inspect repo-google-cloud-java) != '[]' ]]; then
88+
docker volume rm repo-google-cloud-java
89+
fi
90+
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind"
91+
- name: generate from configuration
92+
id: generation
93+
shell: bash
94+
run: |
95+
set -x
96+
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java"
97+
echo "::set-output name=repo_volumes::${repo_volumes}"
98+
docker run --rm \
99+
${repo_volumes} \
100+
-v /tmp:/tmp \
101+
-v /var/run/docker.sock:/var/run/docker.sock \
102+
-e "RUNNING_IN_DOCKER=true" \
103+
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \
104+
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \
105+
python /src/generate_repo.py generate \
106+
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \
107+
--repository-path=/workspace/google-cloud-java \
108+
--target-library-api-shortname=${API_SHORTNAME}
109+
env:
110+
API_SHORTNAME: ${{ github.event.inputs.api_shortname }}
111+
- name: Push to branch and create PR
112+
run: |
113+
set -x
114+
[ -z "`git config user.email`" ] && git config --global user.email "cloud-java-bot@google.com"
115+
[ -z "`git config user.name`" ] && git config --global user.name "cloud-java-bot"
116+
117+
# create and push to branch in origin
118+
# random_id allows multiple runs of this workflow
119+
random_id=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 5; echo)
120+
branch_name="new-library/${{ github.event.inputs.api_shortname }}-${random_id}"
121+
git checkout -b "${branch_name}"
122+
git add --all
123+
commit_message="feat: [${API_SHORTNAME}] new module for ${API_SHORTNAME}"
124+
git commit -m "${commit_message}"
125+
git remote add monorepo https://cloud-java-bot:${GH_TOKEN}@github.com/${{ github.repository }}.git
126+
git fetch -q --unshallow monorepo
127+
git push -f monorepo "${branch_name}"
128+
129+
# create PR
130+
pr_body="Generated by @${USERNAME} via [generate_new_client_hermetic_build.yaml](https://github.com/googleapis/google-cloud-java/actions/workflows/generate_new_client_hermetic_build.yaml)
131+
132+
Command used:
133+
134+
\`\`\`
135+
python generation/new_client_hermetic_build/add-new-client-config.py add-new-client ${GENERATION_ARGUMENTS}
136+
137+
docker run --rm \\
138+
${DOCKER_VOLUMES} \\
139+
-v /tmp:/tmp \\
140+
-v /var/run/docker.sock:/var/run/docker.sock \\
141+
-e \"RUNNING_IN_DOCKER=true\" \\
142+
-e \"REPO_BINDING_VOLUMES=${DOCKER_VOLUMES}\" \\
143+
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \\
144+
python /src/generate_repo.py generate \\
145+
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \\
146+
--repository-path=/workspace/google-cloud-java \\
147+
--target-library-api-shortname=${API_SHORTNAME}
148+
149+
\`\`\`"
150+
gh pr create --title "${commit_message}" --label "owlbot:run" --head "${branch_name}" --body "${pr_body}"
151+
env:
152+
USERNAME: ${{ github.actor }}
153+
API_SHORTNAME: ${{ github.event.inputs.api_shortname }}
154+
GENERATION_ARGUMENTS: ${{ steps.config_generation.outputs.new_library_args }}
155+
DOCKER_VOLUMES: ${{ steps.generation.outputs.repo_volumes }}
156+
GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }}
157+

‎generation/new_client_hermetic_build/README.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# New client generation (GitHub Action)
2+
This new generation workflow enables generation of new libraries by
3+
1. appending a new library to our [generation_config.yaml](https://github.com/googleapis/google-cloud-java/blob/c7429c0eec419c01d4e2fe14d063b9335efb810b/generation_config.yaml).
4+
2. running the hermetic build scripts docker image and
5+
generate the newly added library.
6+
3. create a PR with the changes.
7+
8+
9+
## Components
10+
### `new_library_hermetic_build/add-new-client-config.py`
11+
This script takes 10 arguments that map to items in the newly added library that
12+
goes in `generation_config.yaml`. A new entry will be added to `libraries` with
13+
the necessary generation configuration.
14+
15+
### `.github/workflows/generate_new_client_hermetic_build.yaml`
16+
This workflow orchestrates the `add-new-client-config.py` script and also uses our docker
17+
image
18+
([gcr.io/cloud-devrel-public-resources/java-library-generation](https://pantheon.corp.google.com/gcr/images/cloud-devrel-public-resources/global/java-library-generation))
19+
to generate a new library. It also creates the pull request.
20+
21+
22+
## Execute the Github Action
23+
24+
In order to run the
25+
[Github Action](https://github.com/googleapis/google-cloud-java/actions/workflows/generate_new_client_hermetic_build.yaml)
26+
, you need to specify a few parameters.
27+
These parameters will be available in the Cloud Drop link (a YAML file) included in the buganizer request.
28+
The example in this README uses AlloyDB's [Cloud Drop](https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1/alloydb_v1.yaml) file as an example.
29+
30+
### API short name (`api_shortname`)
31+
32+
As a convenience for the subsequent commands, we need an identifier for the
33+
library, called `api_shortname`.
34+
This identifier will be used by default to generate the following:
35+
* `--distribution-name`
36+
* `--library_name`
37+
38+
The corresponding value in the Cloud Drop page is `api_short_name`.
39+
40+
Example: `alloydb`
41+
42+
> [!IMPORTANT]
43+
> `api_short_name` is not always unique across client libraries.
44+
> In the instance that the `api_short_name` is already in use by an existing
45+
> client library, you will need to determine a unique name OR to pass a unique
46+
> `library_name`.
47+
> See [Advanced Options](#advanced-options).
48+
49+
### Proto path (`proto_path`)
50+
51+
This is the path from the internal `google3/third_party/googleapis/stable` root to the
52+
directory that contains the proto definitions for a specific version.
53+
For example: `google/datastore/v2`. Root-level proto paths like
54+
`google/datastore` are not supported.
55+
Note that the internal `google3/third_party/googleapis/stable` directory is mirrored externally in https://github.com/googleapis/googleapis/blob/master/.
56+
57+
For example, if the buganizer ticket includes:
58+
59+
> Link to protos: `http://...(omit).../google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`.
60+
61+
then the corresponding external mirrored proto is here: `https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`.
62+
63+
Therefore, the "proto path" value we supply to the command is
64+
`google/cloud/alloydb/v1alpha`.
65+
66+
We will publish a single module for a service that includes the specified version
67+
(in the example, `v1alpha`). Any future version must be manually added to
68+
the configuration yaml (`google-cloud-java/generation_config.yaml`)
69+
70+
#### More than one `proto_path`
71+
72+
If you need another `proto_path` in the library, you must manually add it
73+
to `google-cloud-java/generation_config.yaml` after generating the new client.
74+
75+
### Name pretty (`name_pretty`)
76+
77+
The corresponding value in the Cloud Drop page is `title`.
78+
79+
Example: `AlloyDB API`
80+
81+
### Product Docs (`product_docs`)
82+
83+
The corresponding value in the Cloud Drop page is `documentation_uri`.
84+
The value must starts with "https://".
85+
86+
Example: `https://cloud.google.com/alloydb/docs`
87+
88+
### REST Docs (`rest_docs`)
89+
90+
The corresponding value in the Cloud Drop page is `rest_reference_documentation_uri`.
91+
The value must starts with "https://".
92+
93+
Example: `https://cloud.google.com/alloydb/docs/reference/rest`
94+
95+
If the value exists, add it as a flag to the python command below (see [Advanced
96+
Options](#advanced-options]):
97+
`--rest-docs="https://cloud.google.com/alloydb/docs/reference/rest" \`
98+
99+
### RPC Docs (`rpc_docs`)
100+
101+
The corresponding value in the Cloud Drop page is `proto_reference_documentation_uri`.
102+
The value must starts with "https://".
103+
104+
Example: `https://cloud.google.com/speech-to-text/docs/reference/rpc`
105+
106+
If the value exists, add it as a flag to the python command below (see [Advanced
107+
Options](#advanced-options]):
108+
`--rpc-docs="https://cloud.google.com/speech-to-text/docs/reference/rpc" \`
109+
110+
### API description (`api_description`)
111+
112+
The corresponding value in the Cloud Drop page is `documentation.summary` or `documentation.overview`.
113+
If both of those fields are missing, take the description from the product page above. Use the first sentence to keep it concise.
114+
115+
Example:
116+
```
117+
AlloyDB for PostgreSQL is an open source-compatible database service that
118+
provides a powerful option for migrating, modernizing, or building
119+
commercial-grade applications.
120+
```
121+
122+
### Distribution Name (`distribution_name`)
123+
124+
This variable determines the Maven coordinates of the generated library. It
125+
defaults to `com.google.cloud:google-cloud-{api_shortname}`. This mainly affect
126+
the values in the generated `pom.xml` files.
127+
128+
### Library Name (`library_name`)
129+
130+
This variable indicates the output folder of the library. For example you can
131+
have two libraries with `alloydb` (AlloyDB and AlloyDB Connectors)
132+
as `api_shortname`. In order to avoid both
133+
libraries going to the default `java-alloydb` folder, we can override this
134+
behavior by specifying a value like `alloydb-connectors` so the AlloyDB
135+
Connectors goes to `java-alloydb-connectors`.
136+
137+
## Advanced Options
138+
139+
In case the steps above don't show you how to specify the desired options, you can
140+
run the `add-new-client-config.py` script in your local evironment. The advanced options
141+
not shown in the section above **cannot be specified in the Github Action**,
142+
hence the need for a local run (refer to the "Prerequisites
143+
(for local environment)" section).
144+
For the explanation of the available parameters, run:
145+
`python3.9 generation/new_client_hermetic_build/add-new-client-config.py generate --help`.
146+
147+
After you run the script, you will see that the `generation_config.yaml` file
148+
was modified (or the script exited because the library already existed)
149+
150+
The last step you need is to `cd` into the root of `google-cloud-java` and run
151+
```
152+
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind"
153+
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java"
154+
docker run --rm \
155+
${repo_volumes} \
156+
-v /tmp:/tmp \
157+
-v /var/run/docker.sock:/var/run/docker.sock \
158+
-e "RUNNING_IN_DOCKER=true" \
159+
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \
160+
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \
161+
python /src/generate_repo.py generate \
162+
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \
163+
--repository-path=/workspace/google-cloud-java \
164+
--target-library-api-shortname=<your api_shortname>
165+
166+
```
167+
168+
This docker container will run the generation scripts and generate the library
169+
in your repo. You can create a PR explaining what commands you used (the docker
170+
command is not as informative as the `add-new-client-config.py` call, so make sure at least
171+
the add-new-client-config.py arguments were listed).

0 commit comments

Comments
 (0)