Skip to content

Commit 1aedbf5

Browse files
rahul2393rahul2393
authored
Merge branch 'main' into inline-cache-update-lazy-endpoints
2 parents a4c2cfe + a4997cb commit 1aedbf5

File tree

9 files changed

+344
-6
lines changed

9 files changed

+344
-6
lines changed

‎.agents/skills/new-client-library-generator/SKILL.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
name: new-client-library-generator
3+
description: Generates new Google Cloud Java client libraries by processing service information and updating the hermetic build configuration. Use this skill when tasked with onboarding a new service or version to the google-cloud-java repository.
4+
---
5+
6+
# New Client Library Generator
7+
8+
This skill automates the process of adding a new client library to the `google-cloud-java` repository using the hermetic build system. It retrieves service information from a Buganizer ticket (which links to a Service YAML file) and runs the configuration script to update `generation_config.yaml`.
9+
10+
## Workflow
11+
12+
### 1. Retrieve Service Information from Buganizer
13+
14+
Use the available Buganizer MCP server to fetch the ticket content. The ticket will contain a link to a Service YAML file. Parse this YAML file to extract the fields below.
15+
16+
> [!IMPORTANT]
17+
> Not all fields in the Service YAML file may exist. You must ensure that all **Required** fields are identified and populated before executing the generation script.
18+
19+
**Required:**
20+
- `api_shortname`: Unique service identifier (e.g., `alloydb`).
21+
- `name_pretty`: Human-friendly name (e.g., `AlloyDB API`).
22+
- `proto_path`: Versioned path to protos (e.g., `google/cloud/alloydb/v1`). Must include the version component — root-level paths like `google/cloud/alloydb` are not supported.
23+
- `product_docs`: Product documentation URL (must start with `https://`).
24+
- `api_description`: First sentence of the service summary.
25+
26+
**Optional:**
27+
- `rest_docs`: REST reference documentation URL.
28+
- `rpc_docs`: RPC/proto reference documentation URL.
29+
- `library_name`: Override the default `java-<api_shortname>` directory name.
30+
- `distribution_name`: Override Maven coordinates (default: `com.google.cloud:google-cloud-<api_shortname>`).
31+
32+
For the field-to-flag mapping, see [references/service_yaml_mapping.md](references/service_yaml_mapping.md) (Service YAML fields → script flags).
33+
34+
### 2. Check for Conflicts
35+
36+
Before running the script, verify `api_shortname` is not already in use:
37+
38+
```bash
39+
grep "api_shortname: <API_SHORTNAME>" generation_config.yaml
40+
```
41+
42+
If a conflict exists, determine a unique name or use `--library-name` to set a distinct directory. See [references/generation_guide.md](references/generation_guide.md) for examples.
43+
44+
### 3. Special Cases
45+
46+
Some APIs require non-default Maven coordinates and `api_shortname` values:
47+
48+
| Proto path prefix | `--api-shortname` | `--distribution-name` |
49+
|---------------------|------------------------------|----------------------------------------------------------|
50+
| `google/maps/*` | `maps-<api_short_name>` | `com.google.maps:google-maps-<api_short_name>` |
51+
| `google/shopping/*` | `shopping-<api_short_name>` | `com.google.shopping:google-shopping-<api_short_name>` |
52+
53+
where `<api_short_name>` is the value from the Service YAML.
54+
55+
### 4. Execution
56+
57+
#### Adding a new library
58+
59+
If the library does not exist yet, run the script with the gathered information:
60+
61+
```bash
62+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
63+
--api-shortname="[API_SHORTNAME]" \
64+
--name-pretty="[NAME_PRETTY]" \
65+
--proto-path="[PROTO_PATH]" \
66+
--product-docs="[PRODUCT_DOCS]" \
67+
--api-description="[API_DESCRIPTION]" \
68+
[OPTIONAL_FLAGS]
69+
```
70+
71+
The script modifies `generation_config.yaml` and sorts the `libraries` list alphabetically.
72+
73+
To see all available flags:
74+
```bash
75+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help
76+
```
77+
78+
#### Adding a new version to an existing library
79+
80+
If the client library module already exists and the request is to add a new version, do NOT run the script. Instead, manually add the corresponding `proto_path` for the new version to the `GAPICs` list of the existing library entry in `generation_config.yaml`.
81+
82+
Example entry update:
83+
```yaml
84+
- api_shortname: myapi
85+
...
86+
GAPICs:
87+
- proto_path: google/cloud/myapi/v1
88+
- proto_path: google/cloud/myapi/v2 # Manually added
89+
```
90+
91+
### 5. Verification
92+
93+
After execution or manual update:
94+
1. Confirm `generation_config.yaml` has a new or updated entry with the correct fields. See [references/generation_config_schema.md](references/generation_config_schema.md).
95+
2. Confirm `proto_path` under `GAPICs` includes a version component (e.g., `v1`, `v1beta`, `v1alpha`).
96+
3. Confirm `product_documentation` starts with `https://`.
97+
98+
### 6. Create a Branch and PR
99+
100+
After verifying the changes:
101+
102+
```bash
103+
git checkout -b "new-library/[API_SHORTNAME]"
104+
git add generation_config.yaml
105+
git commit -m "feat: new module for [API_SHORTNAME]"
106+
```
107+
108+
Then open a pull request. The Pull Request body must only contain the `add-new-library` command used.
109+
110+
**PR Body Template:**
111+
112+
```text
113+
Command used:
114+
115+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
116+
--api-shortname="[API_SHORTNAME]" \
117+
--name-pretty="[NAME_PRETTY]" \
118+
--proto-path="[PROTO_PATH]" \
119+
--product-docs="[PRODUCT_DOCS]" \
120+
--api-description="[API_DESCRIPTION]" \
121+
[OPTIONAL_FLAGS]
122+
```
123+
124+
The hermetic library generation workflow will be triggered automatically upon changes to `generation_config.yaml`.

‎.agents/skills/new-client-library-generator/references/generation_config_schema.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# generation_config.yaml Schema
2+
3+
The `generation_config.yaml` file at the root of the repository controls the generation of all client libraries.
4+
5+
## Key Fields
6+
7+
- `gapic_generator_version`: The version of the GAPIC generator used globally (unless overridden).
8+
- `googleapis_commitish`: The commit of the `googleapis` repository used as the source for protos.
9+
- `libraries`: A list of library configurations.
10+
11+
### Library Configuration Fields
12+
13+
**Required (always present):**
14+
- `api_shortname`: (String) Identifier (e.g., `alloydb`).
15+
- `name_pretty`: (String) Display name (e.g., `AlloyDB API`).
16+
- `product_documentation`: (URL) Product documentation URL.
17+
- `api_description`: (String) Service description.
18+
- `client_documentation`: (URL) Auto-generated link to the generated client docs.
19+
- `release_level`: (String) Usually `preview` for new libraries.
20+
- `distribution_name`: (String) Maven coordinates (e.g., `com.google.cloud:google-cloud-alloydb`).
21+
- `api_id`: (String) API identifier (e.g., `alloydb.googleapis.com`).
22+
- `library_type`: (String) Usually `GAPIC_AUTO`.
23+
- `group_id`: (String) Maven group ID (e.g., `com.google.cloud`).
24+
- `cloud_api`: (Boolean) `true` if distribution name starts with `google-cloud-`.
25+
- `GAPICs`: (List) List of proto paths to generate from.
26+
- `proto_path`: (String) Path to versioned protos (must include version, e.g., `google/cloud/alloydb/v1`).
27+
28+
**Optional:**
29+
- `library_name`: (String) Override the output directory name (without `java-` prefix).
30+
- `rest_documentation`: (URL) REST reference documentation URL.
31+
- `rpc_documentation`: (URL) RPC/proto reference documentation URL.
32+
- `requires_billing`: (Boolean) Whether billing is required (default: `true`).
33+
- `api_reference`: (URL) API reference documentation link.
34+
- `codeowner_team`: (String) GitHub team responsible for this library.
35+
- `googleapis_commitish`: (String) Pin to a specific `googleapis/googleapis` commit (overrides repo-level setting).
36+
- `issue_tracker`: (URL) Issue tracker for this library.
37+
- `extra_versioned_modules`: (String) Extra modules managed via `versions.txt`.
38+
- `excluded_dependencies`: (String) Comma-separated dependencies excluded from postprocessing.
39+
- `excluded_poms`: (String) Comma-separated pom files excluded from postprocessing.
40+
41+
## Example Library Entry
42+
43+
```yaml
44+
- api_shortname: alloydb
45+
name_pretty: AlloyDB API
46+
product_documentation: https://cloud.google.com/alloydb/docs
47+
api_description: AlloyDB for PostgreSQL is an open source-compatible database service.
48+
client_documentation: https://cloud.google.com/java/docs/reference/google-cloud-alloydb/latest/overview
49+
release_level: preview
50+
distribution_name: com.google.cloud:google-cloud-alloydb
51+
api_id: alloydb.googleapis.com
52+
library_type: GAPIC_AUTO
53+
group_id: com.google.cloud
54+
cloud_api: true
55+
GAPICs:
56+
- proto_path: google/cloud/alloydb/v1
57+
```

‎.agents/skills/new-client-library-generator/references/generation_guide.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# New Client Library Generation Guide
2+
3+
This guide describes how to generate a new client library by:
4+
1. Appending a new library configuration to `generation_config.yaml` using the provided scripts.
5+
2. Relying on the automated GitHub Actions to generate the code once the configuration is pushed in a Pull Request.
6+
7+
## Components
8+
9+
### `generation/new_client_hermetic_build/add-new-client-config.py`
10+
This script updates `generation_config.yaml` with the necessary metadata for a new library. It takes several arguments that map to fields in the configuration file.
11+
12+
## Local Execution
13+
14+
To add a new library configuration, you must run the `add-new-client-config.py` script. The parameters for this script are typically found in the **Service YAML** file linked in the Buganizer request.
15+
16+
### Basic Usage
17+
18+
```bash
19+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
20+
--api-shortname="[API_SHORTNAME]" \
21+
--name-pretty="[NAME_PRETTY]" \
22+
--proto-path="[PROTO_PATH]" \
23+
--product-docs="[PRODUCT_DOCS]" \
24+
--api-description="[API_DESCRIPTION]"
25+
```
26+
27+
### Parameter Details
28+
29+
#### API Short Name (`--api-shortname`)
30+
The unique identifier for the library (e.g., `alloydb`). This is used to determine the directory name and default artifact name.
31+
- **Source:** `api_short_name` in the Service YAML.
32+
33+
#### Proto Path (`--proto-path`)
34+
The path from the root of the `googleapis` repository to the directory containing the versioned proto definitions.
35+
- **Example:** `google/cloud/alloydb/v1beta`
36+
- **Note:** Root-level paths like `google/cloud/alloydb` are not supported.
37+
38+
#### Name Pretty (`--name-pretty`)
39+
The human-friendly name of the API.
40+
- **Source:** `title` in the Service YAML.
41+
- **Example:** `AlloyDB API`
42+
43+
#### Product Docs (`--product-docs`)
44+
The URL for the product documentation. Must start with `https://`.
45+
- **Source:** `documentation_uri` in the Service YAML.
46+
47+
#### API Description (`--api-description`)
48+
A concise summary of the API for the README.
49+
- **Source:** `documentation.summary` or `documentation.overview` in the Service YAML. Use the first sentence.
50+
51+
## Prerequisites
52+
53+
Ensure you have the following set up in your local environment:
54+
55+
1. **Python 3.9+**: The scripts require Python 3.9 or higher.
56+
2. **Dependencies**: Install the required Python packages from the root of the repository:
57+
```bash
58+
pip install -r generation/new_client_hermetic_build/requirements.txt
59+
```
60+
61+
## Advanced Options
62+
63+
For full control over the generation (e.g., overriding Maven coordinates or library names), run the script with the `--help` flag:
64+
65+
```bash
66+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help
67+
```
68+
69+
### Special Case: Non-Cloud APIs
70+
Some libraries (like Google Maps or Shopping) require special handling for Maven coordinates.
71+
72+
| API Path Prefix | `--api-shortname` | `--distribution-name` |
73+
|---|---|---|
74+
| `google/maps/*` | `maps-<shortname>` | `com.google.maps:google-maps-<shortname>` |
75+
| `google/shopping/*` | `shopping-<shortname>` | `com.google.shopping:google-shopping-<shortname>` |

‎.agents/skills/new-client-library-generator/references/service_yaml_mapping.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Service YAML Mapping to Generation Script Parameters
2+
3+
This document explains how to extract information from a Service YAML file (linked in a Buganizer ticket) to populate the parameters for `generation/new_client_hermetic_build/add-new-client-config.py`.
4+
5+
| Service YAML Field | Script Parameter (`--flag`) | Description |
6+
|---|---|---|
7+
| `api_short_name` | `api-shortname` | Unique identifier for the service (e.g., `alloydb`). |
8+
| (extracted from proto) | `proto-path` | Path from the root of the googleapis repository to the versioned proto directory (e.g., `google/cloud/alloydb/v1alpha`). Root-level paths like `google/cloud/alloydb` are not supported. |
9+
| `title` | `name-pretty` | Human-friendly name (e.g., `AlloyDB API`). |
10+
| `documentation_uri` | `product-docs` | Product documentation URL. Must start with `https://`. |
11+
| `rest_reference_documentation_uri` | `rest-docs` | REST documentation URL. Optional. |
12+
| `proto_reference_documentation_uri` | `rpc-docs` | RPC/Proto documentation URL. Optional. |
13+
| `documentation.summary` or `documentation.overview` | `api-description` | Concise summary of the service. Use the first sentence. |
14+
15+
## Example Mapping
16+
17+
**Service YAML File excerpt:**
18+
```yaml
19+
title: "Discovery Engine API"
20+
api_short_name: "discoveryengine"
21+
documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/introduction"
22+
rest_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rest"
23+
proto_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc"
24+
documentation:
25+
summary: "Discovery Engine for Search and Recommendations"
26+
```
27+
28+
**Protos in `googleapis` repository:**
29+
`google/cloud/discoveryengine/v1/`
30+
31+
**Generated Command:**
32+
```bash
33+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
34+
--api-shortname="discoveryengine" \
35+
--name-pretty="Discovery Engine API" \
36+
--proto-path="google/cloud/discoveryengine/v1" \
37+
--product-docs="https://cloud.google.com/generative-ai-app-builder/docs/introduction" \
38+
--rest-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rest" \
39+
--rpc-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc" \
40+
--api-description="Discovery Engine for Search and Recommendations"
41+
```

‎sdk-platform-java/hermetic_build/library_generation/postprocess_library.sh‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,7 @@ if [[ "${is_monorepo}" == "true" ]]; then
7373
# - "/google-.*/src"
7474

7575
library_name=$(basename "${postprocessing_target}")
76-
cat "${postprocessing_target}/${owlbot_yaml_file_name}" \
77-
| sed "s/- \"\/${library_name}/ - \"/" \
78-
> "${postprocessing_target}/.OwlBot.hermetic.yaml"
76+
normalize_owlbot_yaml "${postprocessing_target}/${owlbot_yaml_file_name}" "${postprocessing_target}/.OwlBot.hermetic.yaml" "${library_name}"
7977
owlbot_yaml_relative_path=".OwlBot.hermetic.yaml"
8078
else
8179
owlbot_yaml_relative_path=".github/${owlbot_yaml_file_name}"

‎sdk-platform-java/hermetic_build/library_generation/templates/owlbot.yaml.monorepo.j2‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ deep-remove-regex:
2121

2222
deep-preserve-regex:
2323
- "/{{ module_name }}/google-.*/src/test/java/com/google/cloud/.*/v.*/it/IT.*Test.java"
24-
- "/.*google-.*/src/main/java/com/google/[^/]+/.*/v.*/stub/Version.java"
24+
- "/.*google-.*/src/main/java/.*/stub/Version.java"
2525

2626
deep-copy-regex:
2727
- source: "/{{ proto_path }}/(v.*)/.*-java/proto-google-.*/src"

‎sdk-platform-java/hermetic_build/library_generation/tests/generate_library_unit_tests.sh‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,35 @@ get_proto_path_from_preprocessed_sources_multiple_proto_dirs_fails() {
213213
assertEquals 1 ${res}
214214
}
215215

216+
normalize_owlbot_yaml_test() {
217+
local temp_dir=$(mktemp -d)
218+
local input_file="${temp_dir}/input.yaml"
219+
local output_file="${temp_dir}/output.yaml"
220+
221+
cat <<EOF > "${input_file}"
222+
deep-remove-regex:
223+
- "/java-accesscontextmanager/proto-google-.*/src"
224+
- "/java-accesscontextmanager/samples/snippets/generated"
225+
- "/.*google-.*/src/main/java/.*/stub/Version.java"
226+
EOF
227+
228+
normalize_owlbot_yaml "${input_file}" "${output_file}" "java-accesscontextmanager"
229+
230+
# Verify content
231+
local expected_content
232+
expected_content=$(cat <<EOF
233+
deep-remove-regex:
234+
- "/proto-google-.*/src"
235+
- "/samples/snippets/generated"
236+
- "/.*google-.*/src/main/java/.*/stub/Version.java"
237+
EOF
238+
)
239+
local actual_content=$(cat "${output_file}")
240+
assertEquals "${expected_content}" "${actual_content}"
241+
242+
rm -rf "${temp_dir}"
243+
}
244+
216245
# Execute tests.
217246
# One line per test.
218247
test_list=(
@@ -237,6 +266,7 @@ test_list=(
237266
get_proto_path_from_preprocessed_sources_valid_library_succeeds
238267
get_proto_path_from_preprocessed_sources_empty_library_fails
239268
get_proto_path_from_preprocessed_sources_multiple_proto_dirs_fails
269+
normalize_owlbot_yaml_test
240270
)
241271

242272
pushd "${script_dir}"

‎sdk-platform-java/hermetic_build/library_generation/tests/resources/goldens/.OwlBot-hermetic-golden.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ deep-remove-regex:
2121

2222
deep-preserve-regex:
2323
- "/java-bare-metal-solution/google-.*/src/test/java/com/google/cloud/.*/v.*/it/IT.*Test.java"
24-
- "/.*google-.*/src/main/java/com/google/[^/]+/.*/v.*/stub/Version.java"
24+
- "/.*google-.*/src/main/java/.*/stub/Version.java"
2525

2626
deep-copy-regex:
2727
- source: "/google/cloud/baremetalsolution/(v.*)/.*-java/proto-google-.*/src"

‎sdk-platform-java/hermetic_build/library_generation/utils/utilities.sh‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ download_protoc() {
108108
"protoc-${protoc_version}.zip" \
109109
"GitHub"
110110
unzip -o -q "protoc-${protoc_version}.zip"
111-
rm "protoc-${protoc_version}.zip" "readme.txt"
111+
rm -f "protoc-${protoc_version}.zip" "readme.txt"
112112
}
113113

114114
download_grpc_plugin() {
@@ -333,3 +333,16 @@ error_if_not_exists() {
333333
exit 1
334334
fi
335335
}
336+
337+
normalize_owlbot_yaml() {
338+
local input_file="$1"
339+
local output_file="$2"
340+
local library_name="$3"
341+
342+
# Step 1: Remove library name prefix and normalize indentation for matching lines.
343+
# Step 2: Normalize indentation for lines that didn't have the prefix.
344+
# Using | as sed delimiter to avoid escaping forward slashes.
345+
sed -e "s|^\s*- \"/${library_name}|- \"|" \
346+
-e "s|^\s*- \"|- \"|" \
347+
"${input_file}" > "${output_file}"
348+
}

0 commit comments

Comments
 (0)