diff --git a/.agents/skills/new-client-library-generator/SKILL.md b/.agents/skills/new-client-library-generator/SKILL.md new file mode 100644 index 000000000000..0de415165391 --- /dev/null +++ b/.agents/skills/new-client-library-generator/SKILL.md @@ -0,0 +1,105 @@ +--- +name: new-client-library-generator +description: Generates new Google Cloud Java client libraries by processing service information and updating the hermetic build configuration. Use this skill when tasked with onboarding a new service or version to the google-cloud-java repository. +--- + +# New Client Library Generator + +This skill automates the process of adding a new client library to the `google-cloud-java` repository using the hermetic build system. It retrieves service information from a Buganizer ticket (which links to a Service YAML file) and runs the configuration script to update `generation_config.yaml`. + +## Workflow + +### 1. Retrieve Service Information from Buganizer + +Use the available Buganizer MCP server to fetch the ticket content. The ticket will contain a link to a Service YAML file. Parse this YAML file to extract the following fields: + +**Required:** +- `api_shortname`: Unique service identifier (e.g., `alloydb`). +- `name_pretty`: Human-friendly name (e.g., `AlloyDB API`). +- `proto_path`: Versioned path to protos (e.g., `google/cloud/alloydb/v1`). Must include the version component — root-level paths like `google/cloud/alloydb` are not supported. +- `product_docs`: Product documentation URL (must start with `https://`). +- `api_description`: First sentence of the service summary. + +**Optional:** +- `rest_docs`: REST reference documentation URL. +- `rpc_docs`: RPC/proto reference documentation URL. +- `library_name`: Override the default `java-` directory name. +- `distribution_name`: Override Maven coordinates (default: `com.google.cloud:google-cloud-`). + +For the field-to-flag mapping, see [references/service_yaml_mapping.md](references/service_yaml_mapping.md) (Service YAML fields → script flags). + +### 2. Check for Conflicts + +Before running the script, verify `api_shortname` is not already in use: + +```bash +grep "api_shortname: " generation_config.yaml +``` + +If a conflict exists, determine a unique name or use `--library-name` to set a distinct directory. See [references/generation_guide.md](references/generation_guide.md) for examples. + +### 3. Special Cases + +Some APIs require non-default Maven coordinates and `api_shortname` values: + +| Proto path prefix | `--api-shortname` | `--distribution-name` | +|---------------------|------------------------------|----------------------------------------------------------| +| `google/maps/*` | `maps-` | `com.google.maps:google-maps-` | +| `google/shopping/*` | `shopping-` | `com.google.shopping:google-shopping-` | + +where `` is the value from the Service YAML. + +### 4. Execution + +#### Adding a new library + +If the library does not exist yet, run the script with the gathered information: + +```bash +python3 generation/new_client_hermetic_build/add-new-client-config.py generate \ + --api-shortname="[API_SHORTNAME]" \ + --name-pretty="[NAME_PRETTY]" \ + --proto-path="[PROTO_PATH]" \ + --product-docs="[PRODUCT_DOCS]" \ + --api-description="[API_DESCRIPTION]" \ + [OPTIONAL_FLAGS] +``` + +The script modifies `generation_config.yaml` and sorts the `libraries` list alphabetically. + +To see all available flags: +```bash +python3 generation/new_client_hermetic_build/add-new-client-config.py generate --help +``` + +#### Adding a new version to an existing library + +If the client library module already exists and the request is to add a new version, do NOT run the script. Instead, manually add the corresponding `proto_path` for the new version to the `GAPICs` list of the existing library entry in `generation_config.yaml`. + +Example entry update: +```yaml +- api_shortname: myapi + ... + GAPICs: + - proto_path: google/cloud/myapi/v1 + - proto_path: google/cloud/myapi/v2 # Manually added +``` + +### 5. Verification + +After execution or manual update: +1. Confirm `generation_config.yaml` has a new or updated entry with the correct fields. See [references/generation_config_schema.md](references/generation_config_schema.md). +2. Confirm `proto_path` under `GAPICs` includes a version component (e.g., `v1`, `v1beta`, `v1alpha`). +3. Confirm `product_documentation` starts with `https://`. + +### 6. Create a Branch and PR + +After verifying the changes: + +```bash +git checkout -b "new-library/[API_SHORTNAME]" +git add generation_config.yaml +git commit -m "feat: [API_SHORTNAME] new module for [API_SHORTNAME]" +``` + +Then open a pull request. Include the exact `generate` command and arguments used (if applicable) in the PR body. The hermetic library generation workflow will be triggered automatically upon changes to `generation_config.yaml`. diff --git a/.agents/skills/new-client-library-generator/references/generation_config_schema.md b/.agents/skills/new-client-library-generator/references/generation_config_schema.md new file mode 100644 index 000000000000..075db9054032 --- /dev/null +++ b/.agents/skills/new-client-library-generator/references/generation_config_schema.md @@ -0,0 +1,57 @@ +# generation_config.yaml Schema + +The `generation_config.yaml` file at the root of the repository controls the generation of all client libraries. + +## Key Fields + +- `gapic_generator_version`: The version of the GAPIC generator used globally (unless overridden). +- `googleapis_commitish`: The commit of the `googleapis` repository used as the source for protos. +- `libraries`: A list of library configurations. + +### Library Configuration Fields + +**Required (always present):** +- `api_shortname`: (String) Identifier (e.g., `alloydb`). +- `name_pretty`: (String) Display name (e.g., `AlloyDB API`). +- `product_documentation`: (URL) Product documentation URL. +- `api_description`: (String) Service description. +- `client_documentation`: (URL) Auto-generated link to the generated client docs. +- `release_level`: (String) Usually `preview` for new libraries. +- `distribution_name`: (String) Maven coordinates (e.g., `com.google.cloud:google-cloud-alloydb`). +- `api_id`: (String) API identifier (e.g., `alloydb.googleapis.com`). +- `library_type`: (String) Usually `GAPIC_AUTO`. +- `group_id`: (String) Maven group ID (e.g., `com.google.cloud`). +- `cloud_api`: (Boolean) `true` if distribution name starts with `google-cloud-`. +- `GAPICs`: (List) List of proto paths to generate from. + - `proto_path`: (String) Path to versioned protos (must include version, e.g., `google/cloud/alloydb/v1`). + +**Optional:** +- `library_name`: (String) Override the output directory name (without `java-` prefix). +- `rest_documentation`: (URL) REST reference documentation URL. +- `rpc_documentation`: (URL) RPC/proto reference documentation URL. +- `requires_billing`: (Boolean) Whether billing is required (default: `true`). +- `api_reference`: (URL) API reference documentation link. +- `codeowner_team`: (String) GitHub team responsible for this library. +- `googleapis_commitish`: (String) Pin to a specific `googleapis/googleapis` commit (overrides repo-level setting). +- `issue_tracker`: (URL) Issue tracker for this library. +- `extra_versioned_modules`: (String) Extra modules managed via `versions.txt`. +- `excluded_dependencies`: (String) Comma-separated dependencies excluded from postprocessing. +- `excluded_poms`: (String) Comma-separated pom files excluded from postprocessing. + +## Example Library Entry + +```yaml +- api_shortname: alloydb + name_pretty: AlloyDB API + product_documentation: https://cloud.google.com/alloydb/docs + api_description: AlloyDB for PostgreSQL is an open source-compatible database service. + client_documentation: https://cloud.google.com/java/docs/reference/google-cloud-alloydb/latest/overview + release_level: preview + distribution_name: com.google.cloud:google-cloud-alloydb + api_id: alloydb.googleapis.com + library_type: GAPIC_AUTO + group_id: com.google.cloud + cloud_api: true + GAPICs: + - proto_path: google/cloud/alloydb/v1 +``` diff --git a/.agents/skills/new-client-library-generator/references/generation_guide.md b/.agents/skills/new-client-library-generator/references/generation_guide.md new file mode 100644 index 000000000000..0561cbde028c --- /dev/null +++ b/.agents/skills/new-client-library-generator/references/generation_guide.md @@ -0,0 +1,266 @@ +# New client generation (GitHub Action) +This new generation workflow enables generation of new libraries by + 1. Appending a new library to our [generation_config.yaml](/generation_config.yaml). + 2. Creating a PR with the changes. + +The new client will be generated by [Hermetic library generation workflow](/.github/workflows/hermetic_library_generation.yaml). + + +## Components +### generation/new_client_hermetic_build/add-new-client-config.py +This script takes 10 arguments that map to items in the newly added library that +goes in `generation_config.yaml`. +A new entry will be added to `libraries` with the necessary generation +configuration. + +### `.github/workflows/generate_new_client_hermetic_build.yaml` +This workflow orchestrates the `add-new-client-config.py` script and creates +a pull request. + + +## Execute the GitHub Action + +In order to run the +[GitHub Action](/.github/workflows/generate_new_client_hermetic_build.yaml), +you need to specify a few parameters. +These parameters will be available in the Cloud Drop link (a YAML file) included +in the Buganizer request. +The example in this README uses AlloyDB's [Cloud Drop](https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1/alloydb_v1.yaml) +file as an example. + + +:warning: **IMPORTANT:** +Not all the `new-client.py` arguments are available in the Github Action. +Please refer to +[this +section](/generation/new_client_hermetic_build/README.md#advanced-options) + +### API short name (`api_shortname`) + +As a convenience for the subsequent commands, we need an identifier for the +library, called `api_shortname`. +This identifier will be used by default to generate the following: +* `--distribution-name` +* --library-name + +The corresponding value in the Cloud Drop page is `api_short_name`. + +Example: `alloydb` + +> [!IMPORTANT] +> `api_short_name` is not always unique across client libraries. +> In the instance that the `api_short_name` is already in use by an existing +> client library, you will need to determine a unique name OR to pass a unique +> `library_name`. +> See [Advanced Options](#advanced-options). + +### Proto path (`proto_path`) + +This is the path from the internal `google3/third_party/googleapis/stable` root +to the directory that contains the proto definitions for a specific version. +For example: `google/datastore/v2`. +Root-level proto paths like `google/datastore` are not supported. +Note that the internal `google3/third_party/googleapis/stable` directory is +mirrored externally in https://github.com/googleapis/googleapis/blob/master/. + +For example, if the Buganizer ticket includes: + +> Link to protos: `http://...(omit).../google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`. + +then the corresponding external mirrored proto is here: `https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`. + +Therefore, the "proto path" value we supply to the command is +`google/cloud/alloydb/v1alpha`. + +We will publish a single module for a service that includes the specified version +(in the example, `v1alpha`). +Any future version must be manually added to the configuration yaml (`google-cloud-java/generation_config.yaml`) + +#### More than one `proto_path` + +If you need another `proto_path` in the library, you must manually add it +to `google-cloud-java/generation_config.yaml` after generating the new client. + +### Name pretty (`name_pretty`) + +The corresponding value in the Cloud Drop page is `title`. + +Example: `AlloyDB API` + +### Product Docs (`product_docs`) + +The corresponding value in the Cloud Drop page is `documentation_uri`. +The value must starts with "https://". + +Example: `https://cloud.google.com/alloydb/docs` + +### REST Docs (`rest_docs`) + +The corresponding value in the Cloud Drop page is `rest_reference_documentation_uri`. +The value must starts with "https://". + +Example: `https://cloud.google.com/alloydb/docs/reference/rest` + +If the value exists, add it as a flag to the python command below (see [Advanced +Options](#advanced-options): +`--rest-docs="https://cloud.google.com/alloydb/docs/reference/rest" \` + +### RPC Docs (`rpc_docs`) + +The corresponding value in the Cloud Drop page is `proto_reference_documentation_uri`. +The value must starts with "https://". + +Example: `https://cloud.google.com/speech-to-text/docs/reference/rpc` + +If the value exists, add it as a flag to the python command below (see [Advanced +Options](#advanced-options): +`--rpc-docs="https://cloud.google.com/speech-to-text/docs/reference/rpc" \` + +### API description (`api_description`) + +The corresponding value in the Cloud Drop page is `documentation.summary` or `documentation.overview`. +If both of those fields are missing, take the description from the product page +above. +Use the first sentence to keep it concise. + +Example: +``` + AlloyDB for PostgreSQL is an open source-compatible database service that + provides a powerful option for migrating, modernizing, or building + commercial-grade applications. + ``` + +### Distribution Name (`distribution_name`) + +This variable determines the Maven coordinates of the generated library. +It defaults to `com.google.cloud:google-cloud-{api_shortname}`. +This mainly affect the values in the generated `pom.xml` files. + +### Library Name (`library_name`) + +This variable indicates the output folder of the library. +For example, you can have two libraries with `alloydb` +(AlloyDB and AlloyDB Connectors) as `api_shortname`. +In order to avoid both libraries going to the default `java-alloydb` folder, +we can override this behavior by specifying a value like `alloydb-connectors` +so the AlloyDB Connectors goes to `java-alloydb-connectors`. + +## Prerequisites (for local environment) + +This section is only needed for the first _local_ run of this script. + +### Checkout google-cloud-java repository + +``` +$ git clone https://github.com/googleapis/google-cloud-java +$ git checkout main +$ git pull +``` + +### Install pyenv + +Install pyenv + +``` +curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer \ +| bash +``` + +Append the following lines to `$HOME/.bashrc`. + +``` +export PYENV_ROOT="$HOME/.pyenv" +command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" +eval "$(pyenv init -)" +eval "$(pyenv virtualenv-init -)" +``` + +Logout the shell and login again. You should be at the home directory. + +Assuming you have the following folder structure: +``` +~ (Home) + -> IdeaProjects/ + -> google-cloud-java + -> ... +``` +You can run these next commands in the home directory (or IdeaProjects). Otherwise, run it at `google-cloud-java`'s parent directory. + +Confirm pyenv installation succeeded: + +``` +~$ pyenv +pyenv 2.3.4 +Usage: pyenv [] + +Some useful pyenv commands are: + activate Activate virtual environment + commands List all available pyenv commands + deactivate Deactivate virtual environment +... +``` + +### Install Python 3.9 via pyenv + +``` +~$ pyenv install 3.9.13 +Downloading Python-3.9.13.tar.xz... +-> https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tar.xz +Installing Python-3.9.13... +WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib? +Installed Python-3.9.13 to /usr/local/google/home//.pyenv/versions/3.9.13 +``` + +### Install Python v3.9.13 locally + +Run this command + +``` +$ pyenv local 3.9.13 +``` + +Confirm `python3.9` is installed: +``` +$ which python3.9 +/usr/local/google/home//.pyenv/shims/python3.9 +``` + +### Install Python packages + +At the root of google-cloud-java repository clone, run: + +``` +$ python3.9 -m pip install -r generation/new_client_hermetic_build/requirements.txt +``` + +## Advanced Options + +In case the steps above don't show you how to specify the desired options, you +can run the `add-new-client-config.py` script in your local environment. +The advanced options not shown in the section above **cannot be specified in +the GitHub Action**, hence the need for a local run (refer to the "Prerequisites +(for local environment)" section). +For the explanation of the available parameters, run: +``` +python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help +``` + +After you run the script, you will see that the `generation_config.yaml` file +was modified (or the script exited because the library already existed). + +Please create a PR explaining what commands you used and make sure the +add-new-client-config.py arguments were listed). + +### Special case example: Google Maps + +Sometimes, a library generation requires special handling for +Maven coordinates or API ID, especially when the library is not +specific to Google Cloud. The table below is the summary of the +special cases: + +| API paths | --api-shortname | --distribution-name | +|-------------------|-----------------------------|--------------------------------------------------------| +| google/shopping/* | shopping- | com.google.shopping:google-shopping- | +| google/maps/* | `maps-` | `com.google.maps:google-maps-` | + +where `` is the value from Cloud Drop file. diff --git a/.agents/skills/new-client-library-generator/references/service_yaml_mapping.md b/.agents/skills/new-client-library-generator/references/service_yaml_mapping.md new file mode 100644 index 000000000000..11aabc0fcf0f --- /dev/null +++ b/.agents/skills/new-client-library-generator/references/service_yaml_mapping.md @@ -0,0 +1,41 @@ +# Service YAML Mapping to Generation Script Parameters + +This document explains how to extract information from a Service YAML file (linked in a Buganizer ticket) to populate the parameters for `generation/new_client_hermetic_build/add-new-client-config.py`. + +| Service YAML Field | Script Parameter (`--flag`) | Description | +|---|---|---| +| `api_short_name` | `api-shortname` | Unique identifier for the service (e.g., `alloydb`). | +| (extracted from proto) | `proto-path` | Path from the root of the googleapis repository to the versioned proto directory (e.g., `google/cloud/alloydb/v1alpha`). Root-level paths like `google/cloud/alloydb` are not supported. | +| `title` | `name-pretty` | Human-friendly name (e.g., `AlloyDB API`). | +| `documentation_uri` | `product-docs` | Product documentation URL. Must start with `https://`. | +| `rest_reference_documentation_uri` | `rest-docs` | REST documentation URL. Optional. | +| `proto_reference_documentation_uri` | `rpc-docs` | RPC/Proto documentation URL. Optional. | +| `documentation.summary` or `documentation.overview` | `api-description` | Concise summary of the service. Use the first sentence. | + +## Example Mapping + +**Service YAML File excerpt:** +```yaml +title: "Discovery Engine API" +api_short_name: "discoveryengine" +documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/introduction" +rest_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rest" +proto_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc" +documentation: + summary: "Discovery Engine for Search and Recommendations" +``` + +**Protos in `googleapis` repository:** +`google/cloud/discoveryengine/v1/` + +**Generated Command:** +```bash +python3 generation/new_client_hermetic_build/add-new-client-config.py generate \ + --api-shortname="discoveryengine" \ + --name-pretty="Discovery Engine API" \ + --proto-path="google/cloud/discoveryengine/v1" \ + --product-docs="https://cloud.google.com/generative-ai-app-builder/docs/introduction" \ + --rest-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rest" \ + --rpc-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc" \ + --api-description="Discovery Engine for Search and Recommendations" +```