Skip to content

Commit 8729826

Browse files
lqiu96lqiu96
committed
Merge branch 'datastore-csm-impl-2' into datastore-csm-impl-3
2 parents da326d5 + a804ca4 commit 8729826

5,841 files changed

Lines changed: 492190 additions & 454989 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/java-development/SKILL.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
name: java-development
3+
description: General guidance on Java development practices, building, testing, and style in the monorepo. Use this skill when working on Java code across the repository.
4+
---
5+
6+
# Java Development Guide
7+
8+
This skill provides general guidelines for Java development inside the monorepo. It covers building, formatting, testing, and style conventions to ensure consistency across modules.
9+
10+
## Workflow
11+
12+
### 1. Building the Project
13+
14+
The repository uses Maven as its primary build system.
15+
16+
* **Build All Modules**: To build all modules from the root of the repository, run:
17+
```bash
18+
mvn install -T 1C -P quick-build
19+
```
20+
> [!TIP]
21+
> Use `-T 1C` to build modules in parallel (one thread per CPU core) and `-P quick-build` to skip unnecessary plugins for faster builds.
22+
* **Build a Specific Module**: You can also run Maven commands within a specific module directory (e.g., `java-bigquery`) to build only that module.
23+
24+
### 2. Code Formatting
25+
26+
Code formatting is enforced using the `fmt-maven-plugin`.
27+
28+
* **Check Formatting**: To check for formatting issues without modifying files, run:
29+
```bash
30+
mvn fmt:check -T 1C
31+
```
32+
* **Apply Formatting**: To automatically format the code according to the project style, run:
33+
```bash
34+
mvn fmt:format -T 1C
35+
```
36+
> [!TIP]
37+
> To save time, run `mvn fmt:format` within the specific module directory you are working on, rather than at the root.
38+
> [!NOTE]
39+
> Always run `mvn fmt:format` before committing changes to avoid build failures due to formatting.
40+
41+
### 3. Testing Strategy
42+
43+
* **Unit Tests**: Traditional unit tests should be added for individual classes and methods. Run them using:
44+
```bash
45+
mvn test -T 1C
46+
```
47+
* **Integration Tests**: Many modules have integration tests that run against live services or emulators. These may require specific profiles or environment variables. Refer to the specific module's README for details.
48+
49+
### 4. Style Guide
50+
51+
Follow these general rules to maintain code quality and consistency:
52+
53+
1. **Minimize Visibility**: Default to the most restrictive access level possible. Avoid using `public` unless the class or method is intended to be part of the public API.
54+
2. Avoid Fully Qualified Names: Use imports to keep class names short and readable, rather than using fully qualified names in the code.
55+
3. **Avoid Obsolete APIs**: Do not call methods marked with `@ObsoleteApi` or `@Deprecated` unless there are no viable alternatives.
56+
4. **Clean Diffs**: Avoid unnecessary formatting changes or whitespace modifications to keep diffs clean and easy to review.
57+
58+
### 5. Dependency Management
59+
60+
* **Version Bumps**: Try not to bump any external dependency versions unless there is a known security vulnerability (CVE) or a critical bug fix.
61+
* **New Dependencies**: Avoid introducing new external dependencies. If a new dependency is required, provide a strong justification in the pull request.
62+
* **Standard Library First**: Prefer to use features from the Java standard library, followed by existing dependencies in the project (preferably Google-managed dependencies).
63+
64+
### 6. Contribution Guidelines
65+
66+
* **Commit Messages**: Follow the [Conventional Commits](https://www.conventionalcommits.org/) specification. Include the module as the scope (e.g., `feat(spanner): ...`, `fix(bigquery): ...`).
67+
* **Pull Requests**: All code changes must be submitted via a pull request and require review. Ensure you pull the latest changes from `main` and resolve any conflicts before submitting.

‎.agents/skills/new-client-library-generator/SKILL.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
name: new-client-library-generator
3+
description: Generates new Google Cloud Java client libraries by processing service information and updating the hermetic build configuration. Use this skill when tasked with onboarding a new service or version to the google-cloud-java repository.
4+
---
5+
6+
# New Client Library Generator
7+
8+
This skill automates the process of adding a new client library to the `google-cloud-java` repository using the hermetic build system. It retrieves service information from a Buganizer ticket (which links to a Service YAML file) and runs the configuration script to update `generation_config.yaml`.
9+
10+
## Workflow
11+
12+
### 1. Retrieve Service Information from Buganizer
13+
14+
Use the available Buganizer MCP server to fetch the ticket content. The ticket will contain a link to a Service YAML file. Parse this YAML file to extract the fields below.
15+
16+
> [!IMPORTANT]
17+
> Not all fields in the Service YAML file may exist. You must ensure that all **Required** fields are identified and populated before executing the generation script.
18+
19+
**Required:**
20+
- `api_shortname`: Unique service identifier (e.g., `alloydb`).
21+
- `name_pretty`: Human-friendly name (e.g., `AlloyDB API`).
22+
- `proto_path`: Versioned path to protos (e.g., `google/cloud/alloydb/v1`). Must include the version component — root-level paths like `google/cloud/alloydb` are not supported.
23+
- `product_docs`: Product documentation URL (must start with `https://`).
24+
- `api_description`: First sentence of the service summary.
25+
26+
**Optional:**
27+
- `rest_docs`: REST reference documentation URL.
28+
- `rpc_docs`: RPC/proto reference documentation URL.
29+
- `library_name`: Override the default `java-<api_shortname>` directory name.
30+
- `distribution_name`: Override Maven coordinates (default: `com.google.cloud:google-cloud-<api_shortname>`).
31+
32+
For the field-to-flag mapping, see [references/service_yaml_mapping.md](references/service_yaml_mapping.md) (Service YAML fields → script flags).
33+
34+
### 2. Check for Conflicts
35+
36+
Before running the script, verify `api_shortname` is not already in use:
37+
38+
```bash
39+
grep "api_shortname: <API_SHORTNAME>" generation_config.yaml
40+
```
41+
42+
If a conflict exists, determine a unique name or use `--library-name` to set a distinct directory. See [references/generation_guide.md](references/generation_guide.md) for examples.
43+
44+
### 3. Special Cases
45+
46+
Some APIs require non-default Maven coordinates and `api_shortname` values:
47+
48+
| Proto path prefix | `--api-shortname` | `--distribution-name` |
49+
|---------------------|------------------------------|----------------------------------------------------------|
50+
| `google/maps/*` | `maps-<api_short_name>` | `com.google.maps:google-maps-<api_short_name>` |
51+
| `google/shopping/*` | `shopping-<api_short_name>` | `com.google.shopping:google-shopping-<api_short_name>` |
52+
53+
where `<api_short_name>` is the value from the Service YAML.
54+
55+
### 4. Execution
56+
57+
#### Adding a new library
58+
59+
If the library does not exist yet, run the script with the gathered information:
60+
61+
```bash
62+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
63+
--api-shortname="[API_SHORTNAME]" \
64+
--name-pretty="[NAME_PRETTY]" \
65+
--proto-path="[PROTO_PATH]" \
66+
--product-docs="[PRODUCT_DOCS]" \
67+
--api-description="[API_DESCRIPTION]" \
68+
[OPTIONAL_FLAGS]
69+
```
70+
71+
The script modifies `generation_config.yaml` and sorts the `libraries` list alphabetically.
72+
73+
To see all available flags:
74+
```bash
75+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help
76+
```
77+
78+
#### Adding a new version to an existing library
79+
80+
If the client library module already exists and the request is to add a new version, do NOT run the script. Instead, manually add the corresponding `proto_path` for the new version to the `GAPICs` list of the existing library entry in `generation_config.yaml`.
81+
82+
Example entry update:
83+
```yaml
84+
- api_shortname: myapi
85+
...
86+
GAPICs:
87+
- proto_path: google/cloud/myapi/v1
88+
- proto_path: google/cloud/myapi/v2 # Manually added
89+
```
90+
91+
### 5. Verification
92+
93+
After execution or manual update:
94+
1. Confirm `generation_config.yaml` has a new or updated entry with the correct fields. See [references/generation_config_schema.md](references/generation_config_schema.md).
95+
2. Confirm `proto_path` under `GAPICs` includes a version component (e.g., `v1`, `v1beta`, `v1alpha`).
96+
3. Confirm `product_documentation` starts with `https://`.
97+
98+
### 6. Create a Branch and PR
99+
100+
After verifying the changes:
101+
102+
```bash
103+
git checkout -b "new-library/[API_SHORTNAME]"
104+
git add generation_config.yaml
105+
git commit -m "feat: new module for [API_SHORTNAME]"
106+
```
107+
108+
Then open a pull request. The Pull Request body must only contain the `add-new-library` command used.
109+
110+
**PR Body Template:**
111+
112+
```text
113+
Command used:
114+
115+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
116+
--api-shortname="[API_SHORTNAME]" \
117+
--name-pretty="[NAME_PRETTY]" \
118+
--proto-path="[PROTO_PATH]" \
119+
--product-docs="[PRODUCT_DOCS]" \
120+
--api-description="[API_DESCRIPTION]" \
121+
[OPTIONAL_FLAGS]
122+
```
123+
124+
The hermetic library generation workflow will be triggered automatically upon changes to `generation_config.yaml`.

‎.agents/skills/new-client-library-generator/references/generation_config_schema.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# generation_config.yaml Schema
2+
3+
The `generation_config.yaml` file at the root of the repository controls the generation of all client libraries.
4+
5+
## Key Fields
6+
7+
- `gapic_generator_version`: The version of the GAPIC generator used globally (unless overridden).
8+
- `googleapis_commitish`: The commit of the `googleapis` repository used as the source for protos.
9+
- `libraries`: A list of library configurations.
10+
11+
### Library Configuration Fields
12+
13+
**Required (always present):**
14+
- `api_shortname`: (String) Identifier (e.g., `alloydb`).
15+
- `name_pretty`: (String) Display name (e.g., `AlloyDB API`).
16+
- `product_documentation`: (URL) Product documentation URL.
17+
- `api_description`: (String) Service description.
18+
- `client_documentation`: (URL) Auto-generated link to the generated client docs.
19+
- `release_level`: (String) Usually `preview` for new libraries.
20+
- `distribution_name`: (String) Maven coordinates (e.g., `com.google.cloud:google-cloud-alloydb`).
21+
- `api_id`: (String) API identifier (e.g., `alloydb.googleapis.com`).
22+
- `library_type`: (String) Usually `GAPIC_AUTO`.
23+
- `group_id`: (String) Maven group ID (e.g., `com.google.cloud`).
24+
- `cloud_api`: (Boolean) `true` if distribution name starts with `google-cloud-`.
25+
- `GAPICs`: (List) List of proto paths to generate from.
26+
- `proto_path`: (String) Path to versioned protos (must include version, e.g., `google/cloud/alloydb/v1`).
27+
28+
**Optional:**
29+
- `library_name`: (String) Override the output directory name (without `java-` prefix).
30+
- `rest_documentation`: (URL) REST reference documentation URL.
31+
- `rpc_documentation`: (URL) RPC/proto reference documentation URL.
32+
- `requires_billing`: (Boolean) Whether billing is required (default: `true`).
33+
- `api_reference`: (URL) API reference documentation link.
34+
- `codeowner_team`: (String) GitHub team responsible for this library.
35+
- `googleapis_commitish`: (String) Pin to a specific `googleapis/googleapis` commit (overrides repo-level setting).
36+
- `issue_tracker`: (URL) Issue tracker for this library.
37+
- `extra_versioned_modules`: (String) Extra modules managed via `versions.txt`.
38+
- `excluded_dependencies`: (String) Comma-separated dependencies excluded from postprocessing.
39+
- `excluded_poms`: (String) Comma-separated pom files excluded from postprocessing.
40+
41+
## Example Library Entry
42+
43+
```yaml
44+
- api_shortname: alloydb
45+
name_pretty: AlloyDB API
46+
product_documentation: https://cloud.google.com/alloydb/docs
47+
api_description: AlloyDB for PostgreSQL is an open source-compatible database service.
48+
client_documentation: https://cloud.google.com/java/docs/reference/google-cloud-alloydb/latest/overview
49+
release_level: preview
50+
distribution_name: com.google.cloud:google-cloud-alloydb
51+
api_id: alloydb.googleapis.com
52+
library_type: GAPIC_AUTO
53+
group_id: com.google.cloud
54+
cloud_api: true
55+
GAPICs:
56+
- proto_path: google/cloud/alloydb/v1
57+
```

‎.agents/skills/new-client-library-generator/references/generation_guide.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# New Client Library Generation Guide
2+
3+
This guide describes how to generate a new client library by:
4+
1. Appending a new library configuration to `generation_config.yaml` using the provided scripts.
5+
2. Relying on the automated GitHub Actions to generate the code once the configuration is pushed in a Pull Request.
6+
7+
## Components
8+
9+
### `generation/new_client_hermetic_build/add-new-client-config.py`
10+
This script updates `generation_config.yaml` with the necessary metadata for a new library. It takes several arguments that map to fields in the configuration file.
11+
12+
## Local Execution
13+
14+
To add a new library configuration, you must run the `add-new-client-config.py` script. The parameters for this script are typically found in the **Service YAML** file linked in the Buganizer request.
15+
16+
### Basic Usage
17+
18+
```bash
19+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
20+
--api-shortname="[API_SHORTNAME]" \
21+
--name-pretty="[NAME_PRETTY]" \
22+
--proto-path="[PROTO_PATH]" \
23+
--product-docs="[PRODUCT_DOCS]" \
24+
--api-description="[API_DESCRIPTION]"
25+
```
26+
27+
### Parameter Details
28+
29+
#### API Short Name (`--api-shortname`)
30+
The unique identifier for the library (e.g., `alloydb`). This is used to determine the directory name and default artifact name.
31+
- **Source:** `api_short_name` in the Service YAML.
32+
33+
#### Proto Path (`--proto-path`)
34+
The path from the root of the `googleapis` repository to the directory containing the versioned proto definitions.
35+
- **Example:** `google/cloud/alloydb/v1beta`
36+
- **Note:** Root-level paths like `google/cloud/alloydb` are not supported.
37+
38+
#### Name Pretty (`--name-pretty`)
39+
The human-friendly name of the API.
40+
- **Source:** `title` in the Service YAML.
41+
- **Example:** `AlloyDB API`
42+
43+
#### Product Docs (`--product-docs`)
44+
The URL for the product documentation. Must start with `https://`.
45+
- **Source:** `documentation_uri` in the Service YAML.
46+
47+
#### API Description (`--api-description`)
48+
A concise summary of the API for the README.
49+
- **Source:** `documentation.summary` or `documentation.overview` in the Service YAML. Use the first sentence.
50+
51+
## Prerequisites
52+
53+
Ensure you have the following set up in your local environment:
54+
55+
1. **Python 3.9+**: The scripts require Python 3.9 or higher.
56+
2. **Dependencies**: Install the required Python packages from the root of the repository:
57+
```bash
58+
pip install -r generation/new_client_hermetic_build/requirements.txt
59+
```
60+
61+
## Advanced Options
62+
63+
For full control over the generation (e.g., overriding Maven coordinates or library names), run the script with the `--help` flag:
64+
65+
```bash
66+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help
67+
```
68+
69+
### Special Case: Non-Cloud APIs
70+
Some libraries (like Google Maps or Shopping) require special handling for Maven coordinates.
71+
72+
| API Path Prefix | `--api-shortname` | `--distribution-name` |
73+
|---|---|---|
74+
| `google/maps/*` | `maps-<shortname>` | `com.google.maps:google-maps-<shortname>` |
75+
| `google/shopping/*` | `shopping-<shortname>` | `com.google.shopping:google-shopping-<shortname>` |

‎.agents/skills/new-client-library-generator/references/service_yaml_mapping.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Service YAML Mapping to Generation Script Parameters
2+
3+
This document explains how to extract information from a Service YAML file (linked in a Buganizer ticket) to populate the parameters for `generation/new_client_hermetic_build/add-new-client-config.py`.
4+
5+
| Service YAML Field | Script Parameter (`--flag`) | Description |
6+
|---|---|---|
7+
| `api_short_name` | `api-shortname` | Unique identifier for the service (e.g., `alloydb`). |
8+
| (extracted from proto) | `proto-path` | Path from the root of the googleapis repository to the versioned proto directory (e.g., `google/cloud/alloydb/v1alpha`). Root-level paths like `google/cloud/alloydb` are not supported. |
9+
| `title` | `name-pretty` | Human-friendly name (e.g., `AlloyDB API`). |
10+
| `documentation_uri` | `product-docs` | Product documentation URL. Must start with `https://`. |
11+
| `rest_reference_documentation_uri` | `rest-docs` | REST documentation URL. Optional. |
12+
| `proto_reference_documentation_uri` | `rpc-docs` | RPC/Proto documentation URL. Optional. |
13+
| `documentation.summary` or `documentation.overview` | `api-description` | Concise summary of the service. Use the first sentence. |
14+
15+
## Example Mapping
16+
17+
**Service YAML File excerpt:**
18+
```yaml
19+
title: "Discovery Engine API"
20+
api_short_name: "discoveryengine"
21+
documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/introduction"
22+
rest_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rest"
23+
proto_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc"
24+
documentation:
25+
summary: "Discovery Engine for Search and Recommendations"
26+
```
27+
28+
**Protos in `googleapis` repository:**
29+
`google/cloud/discoveryengine/v1/`
30+
31+
**Generated Command:**
32+
```bash
33+
python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \
34+
--api-shortname="discoveryengine" \
35+
--name-pretty="Discovery Engine API" \
36+
--proto-path="google/cloud/discoveryengine/v1" \
37+
--product-docs="https://cloud.google.com/generative-ai-app-builder/docs/introduction" \
38+
--rest-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rest" \
39+
--rpc-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc" \
40+
--api-description="Discovery Engine for Search and Recommendations"
41+
```

‎.github/CODEOWNERS‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@
1111
/java-bigquery/ @googleapis/bigquery-team @googleapis/cloud-sdk-java-team
1212
/java-spanner/ @googleapis/spanner-team @googleapis/cloud-sdk-java-team
1313
/java-spanner-jdbc/ @googleapis/spanner-team @googleapis/cloud-sdk-java-team
14-
/google-auth-library-java/ @googleapis/cloud-sdk-auth-team @googleapis/cloud-sdk-java-team
14+
/google-auth-library-java/ @googleapis/cloud-sdk-auth-team @googleapis/cloud-sdk-java-team @googleapis/aion-team
1515
/java-storage/ @googleapis/gcs-team @googleapis/cloud-sdk-java-team
16+
/java-storage-nio/ @googleapis/gcs-team @googleapis/cloud-sdk-java-team

‎.github/header-checker-lint.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
allowedCopyrightHolders:
22
- 'Google LLC'
3+
- 'Google Inc'
34
- 'The Grafeas Authors'

0 commit comments

Comments
 (0)