|
1 | | -# New client generation (GitHub Action) |
2 | | -This new generation workflow enables generation of new libraries by |
3 | | - 1. Appending a new library to our [generation_config.yaml](/generation_config.yaml). |
4 | | - 2. Creating a PR with the changes. |
5 | | - |
6 | | -The new client will be generated by [Hermetic library generation workflow](/.github/workflows/hermetic_library_generation.yaml). |
| 1 | +# New Client Library Generation Guide |
7 | 2 |
|
| 3 | +This guide describes how to generate a new client library by: |
| 4 | +1. Appending a new library configuration to `generation_config.yaml` using the provided scripts. |
| 5 | +2. Relying on the automated GitHub Actions to generate the code once the configuration is pushed in a Pull Request. |
8 | 6 |
|
9 | 7 | ## Components |
10 | | -### generation/new_client_hermetic_build/add-new-client-config.py |
11 | | -This script takes 10 arguments that map to items in the newly added library that |
12 | | -goes in `generation_config.yaml`. |
13 | | -A new entry will be added to `libraries` with the necessary generation |
14 | | -configuration. |
15 | | - |
16 | | -### `.github/workflows/generate_new_client_hermetic_build.yaml` |
17 | | -This workflow orchestrates the `add-new-client-config.py` script and creates |
18 | | -a pull request. |
19 | | - |
20 | | - |
21 | | -## Execute the GitHub Action |
22 | | - |
23 | | -In order to run the |
24 | | -[GitHub Action](/.github/workflows/generate_new_client_hermetic_build.yaml), |
25 | | -you need to specify a few parameters. |
26 | | -These parameters will be available in the Cloud Drop link (a YAML file) included |
27 | | -in the Buganizer request. |
28 | | -The example in this README uses AlloyDB's [Cloud Drop](https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1/alloydb_v1.yaml) |
29 | | -file as an example. |
30 | | - |
31 | | - |
32 | | -:warning: **IMPORTANT:** |
33 | | -Not all the add-new-client-config.py arguments are available in the Github Action. |
34 | | -Please refer to |
35 | | -[this |
36 | | -section](/generation/new_client_hermetic_build/README.md#advanced-options) |
37 | | - |
38 | | -### API short name (`api_shortname`) |
39 | | - |
40 | | -As a convenience for the subsequent commands, we need an identifier for the |
41 | | -library, called `api_shortname`. |
42 | | -This identifier will be used by default to generate the following: |
43 | | -* `--distribution-name` |
44 | | -* --library-name |
45 | | - |
46 | | -The corresponding value in the Cloud Drop page is `api_short_name`. |
47 | | - |
48 | | -Example: `alloydb` |
49 | | - |
50 | | -> [!IMPORTANT] |
51 | | -> `api_short_name` is not always unique across client libraries. |
52 | | -> In the instance that the `api_short_name` is already in use by an existing |
53 | | -> client library, you will need to determine a unique name OR to pass a unique |
54 | | -> `library_name`. |
55 | | -> See [Advanced Options](#advanced-options). |
56 | | -
|
57 | | -### Proto path (`proto_path`) |
58 | | - |
59 | | -This is the path from the internal `google3/third_party/googleapis/stable` root |
60 | | -to the directory that contains the proto definitions for a specific version. |
61 | | -For example: `google/datastore/v2`. |
62 | | -Root-level proto paths like `google/datastore` are not supported. |
63 | | -Note that the internal `google3/third_party/googleapis/stable` directory is |
64 | | -mirrored externally in https://github.com/googleapis/googleapis/blob/master/. |
65 | | - |
66 | | -For example, if the Buganizer ticket includes: |
67 | | - |
68 | | -> Link to protos: `http://...(omit).../google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`. |
69 | | -
|
70 | | -then the corresponding external mirrored proto is here: `https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`. |
71 | | - |
72 | | -Therefore, the "proto path" value we supply to the command is |
73 | | -`google/cloud/alloydb/v1alpha`. |
74 | | - |
75 | | -We will publish a single module for a service that includes the specified version |
76 | | -(in the example, `v1alpha`). |
77 | | -Any future version must be manually added to the configuration yaml (`google-cloud-java/generation_config.yaml`) |
78 | | - |
79 | | -#### More than one `proto_path` |
80 | | - |
81 | | -If you need another `proto_path` in the library, you must manually add it |
82 | | -to `google-cloud-java/generation_config.yaml` after generating the new client. |
83 | | - |
84 | | -### Name pretty (`name_pretty`) |
85 | | - |
86 | | -The corresponding value in the Cloud Drop page is `title`. |
87 | | - |
88 | | -Example: `AlloyDB API` |
89 | | - |
90 | | -### Product Docs (`product_docs`) |
91 | | - |
92 | | -The corresponding value in the Cloud Drop page is `documentation_uri`. |
93 | | -The value must starts with "https://". |
94 | | - |
95 | | -Example: `https://cloud.google.com/alloydb/docs` |
96 | | - |
97 | | -### REST Docs (`rest_docs`) |
98 | | - |
99 | | -The corresponding value in the Cloud Drop page is `rest_reference_documentation_uri`. |
100 | | -The value must starts with "https://". |
101 | | - |
102 | | -Example: `https://cloud.google.com/alloydb/docs/reference/rest` |
103 | | - |
104 | | -If the value exists, add it as a flag to the python command below (see [Advanced |
105 | | -Options](#advanced-options): |
106 | | -`--rest-docs="https://cloud.google.com/alloydb/docs/reference/rest" \` |
107 | | - |
108 | | -### RPC Docs (`rpc_docs`) |
109 | | - |
110 | | -The corresponding value in the Cloud Drop page is `proto_reference_documentation_uri`. |
111 | | -The value must starts with "https://". |
112 | | - |
113 | | -Example: `https://cloud.google.com/speech-to-text/docs/reference/rpc` |
114 | | - |
115 | | -If the value exists, add it as a flag to the python command below (see [Advanced |
116 | | -Options](#advanced-options): |
117 | | -`--rpc-docs="https://cloud.google.com/speech-to-text/docs/reference/rpc" \` |
118 | | - |
119 | | -### API description (`api_description`) |
120 | | - |
121 | | -The corresponding value in the Cloud Drop page is `documentation.summary` or `documentation.overview`. |
122 | | -If both of those fields are missing, take the description from the product page |
123 | | -above. |
124 | | -Use the first sentence to keep it concise. |
125 | 8 |
|
126 | | -Example: |
127 | | -``` |
128 | | - AlloyDB for PostgreSQL is an open source-compatible database service that |
129 | | - provides a powerful option for migrating, modernizing, or building |
130 | | - commercial-grade applications. |
131 | | - ``` |
| 9 | +### `generation/new_client_hermetic_build/add-new-client-config.py` |
| 10 | +This script updates `generation_config.yaml` with the necessary metadata for a new library. It takes several arguments that map to fields in the configuration file. |
132 | 11 |
|
133 | | -### Distribution Name (`distribution_name`) |
| 12 | +## Local Execution |
134 | 13 |
|
135 | | -This variable determines the Maven coordinates of the generated library. |
136 | | -It defaults to `com.google.cloud:google-cloud-{api_shortname}`. |
137 | | -This mainly affect the values in the generated `pom.xml` files. |
| 14 | +To add a new library configuration, you must run the `add-new-client-config.py` script. The parameters for this script are typically found in the **Service YAML** file linked in the Buganizer request. |
138 | 15 |
|
139 | | -### Library Name (`library_name`) |
| 16 | +### Basic Usage |
140 | 17 |
|
141 | | -This variable indicates the output folder of the library. |
142 | | -For example, you can have two libraries with `alloydb` |
143 | | -(AlloyDB and AlloyDB Connectors) as `api_shortname`. |
144 | | -In order to avoid both libraries going to the default `java-alloydb` folder, |
145 | | -we can override this behavior by specifying a value like `alloydb-connectors` |
146 | | -so the AlloyDB Connectors goes to `java-alloydb-connectors`. |
147 | | - |
148 | | -## Prerequisites (for local environment) |
149 | | - |
150 | | -This section is only needed for the first _local_ run of this script. |
151 | | - |
152 | | -### Checkout google-cloud-java repository |
153 | | - |
154 | | -``` |
155 | | -$ git clone https://github.com/googleapis/google-cloud-java |
156 | | -$ git checkout main |
157 | | -$ git pull |
| 18 | +```bash |
| 19 | +python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \ |
| 20 | + --api-shortname="[API_SHORTNAME]" \ |
| 21 | + --name-pretty="[NAME_PRETTY]" \ |
| 22 | + --proto-path="[PROTO_PATH]" \ |
| 23 | + --product-docs="[PRODUCT_DOCS]" \ |
| 24 | + --api-description="[API_DESCRIPTION]" |
158 | 25 | ``` |
159 | 26 |
|
160 | | -### Install pyenv |
| 27 | +### Parameter Details |
161 | 28 |
|
162 | | -Install pyenv |
| 29 | +#### API Short Name (`--api-shortname`) |
| 30 | +The unique identifier for the library (e.g., `alloydb`). This is used to determine the directory name and default artifact name. |
| 31 | +- **Source:** `api_short_name` in the Service YAML. |
163 | 32 |
|
164 | | -``` |
165 | | -curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer \ |
166 | | -| bash |
167 | | -``` |
| 33 | +#### Proto Path (`--proto-path`) |
| 34 | +The path from the root of the `googleapis` repository to the directory containing the versioned proto definitions. |
| 35 | +- **Example:** `google/cloud/alloydb/v1beta` |
| 36 | +- **Note:** Root-level paths like `google/cloud/alloydb` are not supported. |
168 | 37 |
|
169 | | -Append the following lines to `$HOME/.bashrc`. |
| 38 | +#### Name Pretty (`--name-pretty`) |
| 39 | +The human-friendly name of the API. |
| 40 | +- **Source:** `title` in the Service YAML. |
| 41 | +- **Example:** `AlloyDB API` |
170 | 42 |
|
171 | | -``` |
172 | | -export PYENV_ROOT="$HOME/.pyenv" |
173 | | -command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" |
174 | | -eval "$(pyenv init -)" |
175 | | -eval "$(pyenv virtualenv-init -)" |
176 | | -``` |
| 43 | +#### Product Docs (`--product-docs`) |
| 44 | +The URL for the product documentation. Must start with `https://`. |
| 45 | +- **Source:** `documentation_uri` in the Service YAML. |
177 | 46 |
|
178 | | -Logout the shell and login again. You should be at the home directory. |
| 47 | +#### API Description (`--api-description`) |
| 48 | +A concise summary of the API for the README. |
| 49 | +- **Source:** `documentation.summary` or `documentation.overview` in the Service YAML. Use the first sentence. |
179 | 50 |
|
180 | | -Assuming you have the following folder structure: |
181 | | -``` |
182 | | -~ (Home) |
183 | | - -> IdeaProjects/ |
184 | | - -> google-cloud-java |
185 | | - -> ... |
186 | | -``` |
187 | | -You can run these next commands in the home directory (or IdeaProjects). Otherwise, run it at `google-cloud-java`'s parent directory. |
188 | | - |
189 | | -Confirm pyenv installation succeeded: |
| 51 | +## Prerequisites |
190 | 52 |
|
191 | | -``` |
192 | | -~$ pyenv |
193 | | -pyenv 2.3.4 |
194 | | -Usage: pyenv <command> [<args>] |
195 | | -
|
196 | | -Some useful pyenv commands are: |
197 | | - activate Activate virtual environment |
198 | | - commands List all available pyenv commands |
199 | | - deactivate Deactivate virtual environment |
200 | | -... |
201 | | -``` |
202 | | - |
203 | | -### Install Python 3.9 via pyenv |
204 | | - |
205 | | -``` |
206 | | -~$ pyenv install 3.9.13 |
207 | | -Downloading Python-3.9.13.tar.xz... |
208 | | --> https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tar.xz |
209 | | -Installing Python-3.9.13... |
210 | | -WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib? |
211 | | -Installed Python-3.9.13 to /usr/local/google/home/<user>/.pyenv/versions/3.9.13 |
212 | | -``` |
| 53 | +Ensure you have the following set up in your local environment: |
213 | 54 |
|
214 | | -### Install Python v3.9.13 locally |
215 | | - |
216 | | -Run this command |
217 | | - |
218 | | -``` |
219 | | -$ pyenv local 3.9.13 |
220 | | -``` |
221 | | - |
222 | | -Confirm `python3.9` is installed: |
223 | | -``` |
224 | | -$ which python3.9 |
225 | | -/usr/local/google/home/<user>/.pyenv/shims/python3.9 |
226 | | -``` |
227 | | - |
228 | | -### Install Python packages |
229 | | - |
230 | | -At the root of google-cloud-java repository clone, run: |
231 | | - |
232 | | -``` |
233 | | -$ python3.9 -m pip install -r generation/new_client_hermetic_build/requirements.txt |
234 | | -``` |
| 55 | +1. **Python 3.9+**: The scripts require Python 3.9 or higher. |
| 56 | +2. **Dependencies**: Install the required Python packages from the root of the repository: |
| 57 | + ```bash |
| 58 | + pip install -r generation/new_client_hermetic_build/requirements.txt |
| 59 | + ``` |
235 | 60 |
|
236 | 61 | ## Advanced Options |
237 | 62 |
|
238 | | -In case the steps above don't show you how to specify the desired options, you |
239 | | -can run the `add-new-client-config.py` script in your local environment. |
240 | | -The advanced options not shown in the section above **cannot be specified in |
241 | | -the GitHub Action**, hence the need for a local run (refer to the "Prerequisites |
242 | | -(for local environment)" section). |
243 | | -For the explanation of the available parameters, run: |
244 | | -``` |
| 63 | +For full control over the generation (e.g., overriding Maven coordinates or library names), run the script with the `--help` flag: |
| 64 | + |
| 65 | +```bash |
245 | 66 | python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help |
246 | 67 | ``` |
247 | 68 |
|
248 | | -After you run the script, you will see that the `generation_config.yaml` file |
249 | | -was modified (or the script exited because the library already existed). |
250 | | - |
251 | | -Please create a PR explaining what commands you used and make sure the |
252 | | -add-new-client-config.py arguments were listed). |
253 | | - |
254 | | -### Special case example: Google Maps |
255 | | - |
256 | | -Sometimes, a library generation requires special handling for |
257 | | -Maven coordinates or API ID, especially when the library is not |
258 | | -specific to Google Cloud. The table below is the summary of the |
259 | | -special cases: |
260 | | - |
261 | | -| API paths | --api-shortname | --distribution-name | |
262 | | -|-------------------|-----------------------------|--------------------------------------------------------| |
263 | | -| google/shopping/* | shopping-<API short name> | com.google.shopping:google-shopping-<API short name> | |
264 | | -| google/maps/* | `maps-<API short name>` | `com.google.maps:google-maps-<API short name>` | |
| 69 | +### Special Case: Non-Cloud APIs |
| 70 | +Some libraries (like Google Maps or Shopping) require special handling for Maven coordinates. |
265 | 71 |
|
266 | | -where `<API short name>` is the value from Cloud Drop file. |
| 72 | +| API Path Prefix | `--api-shortname` | `--distribution-name` | |
| 73 | +|---|---|---| |
| 74 | +| `google/maps/*` | `maps-<shortname>` | `com.google.maps:google-maps-<shortname>` | |
| 75 | +| `google/shopping/*` | `shopping-<shortname>` | `com.google.shopping:google-shopping-<shortname>` | |
0 commit comments