Merge branch 'main' into main

Author: rahul2393

.agents/skills/java-development/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: java-development
+description: General guidance on Java development practices, building, testing, and style in the monorepo. Use this skill when working on Java code across the repository.
+---
+
+# Java Development Guide
+
+This skill provides general guidelines for Java development inside the monorepo. It covers building, formatting, testing, and style conventions to ensure consistency across modules.
+
+## Workflow
+
+### 1. Building the Project
+
+The repository uses Maven as its primary build system.
+
+*   **Build All Modules**: To build all modules from the root of the repository, run:
+    ```bash
+    mvn install -T 1C -P quick-build
+    ```
+    > [!TIP]
+    > Use `-T 1C` to build modules in parallel (one thread per CPU core) and `-P quick-build` to skip unnecessary plugins for faster builds.
+*   **Build a Specific Module**: You can also run Maven commands within a specific module directory (e.g., `java-bigquery`) to build only that module.
+
+### 2. Code Formatting
+
+Code formatting is enforced using the `fmt-maven-plugin`.
+
+*   **Check Formatting**: To check for formatting issues without modifying files, run:
+    ```bash
+    mvn fmt:check -T 1C
+    ```
+*   **Apply Formatting**: To automatically format the code according to the project style, run:
+    ```bash
+    mvn fmt:format -T 1C
+    ```
+    > [!TIP]
+    > To save time, run `mvn fmt:format` within the specific module directory you are working on, rather than at the root.
+    > [!NOTE]
+    > Always run `mvn fmt:format` before committing changes to avoid build failures due to formatting.
+
+### 3. Testing Strategy
+
+*   **Unit Tests**: Traditional unit tests should be added for individual classes and methods. Run them using:
+    ```bash
+    mvn test -T 1C
+    ```
+*   **Integration Tests**: Many modules have integration tests that run against live services or emulators. These may require specific profiles or environment variables. Refer to the specific module's README for details.
+
+### 4. Style Guide
+
+Follow these general rules to maintain code quality and consistency:
+
+1.  **Minimize Visibility**: Default to the most restrictive access level possible. Avoid using `public` unless the class or method is intended to be part of the public API.
+2.  Avoid Fully Qualified Names: Use imports to keep class names short and readable, rather than using fully qualified names in the code.
+3.  **Avoid Obsolete APIs**: Do not call methods marked with `@ObsoleteApi` or `@Deprecated` unless there are no viable alternatives.
+4.  **Clean Diffs**: Avoid unnecessary formatting changes or whitespace modifications to keep diffs clean and easy to review.
+
+### 5. Dependency Management
+
+*   **Version Bumps**: Try not to bump any external dependency versions unless there is a known security vulnerability (CVE) or a critical bug fix.
+*   **New Dependencies**: Avoid introducing new external dependencies. If a new dependency is required, provide a strong justification in the pull request.
+*   **Standard Library First**: Prefer to use features from the Java standard library, followed by existing dependencies in the project (preferably Google-managed dependencies).
+
+### 6. Contribution Guidelines
+
+*   **Commit Messages**: Follow the [Conventional Commits](https://www.conventionalcommits.org/) specification. Include the module as the scope (e.g., `feat(spanner): ...`, `fix(bigquery): ...`).
+*   **Pull Requests**: All code changes must be submitted via a pull request and require review. Ensure you pull the latest changes from `main` and resolve any conflicts before submitting.

.github/release-note-generation/split_release_note.py

@@ -69,6 +69,7 @@ def detect_modules(root_directory: Path):
         tree = ET.parse(module_pom_xml)
         root = tree.getroot()
         version = root.find('mvn:version', POM_NAMESPACES).text
+        api_name = None
         if owlbot_yaml_path.exists():
             # If OwlBot configuration file exists (most cases), it's the better
             # source to get the OwlBot-generated pull request title prefix than
@@ -78,21 +79,21 @@ def detect_modules(root_directory: Path):
                 match = re.search(r'api-name: (.+)', owlbot_yaml_content)
                 if match:
                     api_name = match.group(1)
-                    modules.append(LibraryModule(module_path, api_name,
-                                                 version,
-                                                 changelog))
+
+        if not api_name:
+            # Fallback to repo-metadata.json (e.g. for vertexai or Spanner transitional state)
+            if repo_metadata_path.exists():
+                with open(repo_metadata_path, 'r') as file:
+                    repo_metadata = json.load(file)
+                    api_name = repo_metadata.get('api_shortname')
+
+        if api_name:
+            modules.append(LibraryModule(module_path, api_name,
+                                         version,
+                                         changelog))
         else:
-            # vertexai (handwritten) does not have OwlBot yaml file
-            with open(repo_metadata_path, 'r') as file:
-                repo_metadata = json.load(file)
-                api_name = repo_metadata['api_shortname']
-                if api_name:
-                    modules.append(LibraryModule(repo_metadata_path.parent, api_name,
-                                                 version,
-                                                 changelog))
-                else:
-                    raise Exception(f'repo_metadata_path {repo_metadata_path} does'
-                                    f' not have api_shortname field')
+            raise Exception(f'Could not determine api-name for {repo_metadata_path}')
+
 
     return modules
 
@@ -133,7 +134,7 @@ def group_changes_by_api(main_changes: [str]):
             elif section == BUG_FIXES_SECTION:
                 api_to_changelog[api_name].bug_fixes.append(note)
             elif section == DEPENDENCIES_SECTION:
-                api_to_changelog[api_name].dependencies.append(note)
+                api_to_changelog[api_name].dependency_upgrades.append(note)
     return api_to_changelog
 
 

.github/release-note-generation/unit_test.py

@@ -1,9 +1,11 @@
 import unittest
+import tempfile
+import json
+from pathlib import Path
 
 # Unit tests for split_release_note.py
 
-from split_release_note import LibraryModule, create_changelog_entry, group_changes_by_api, ChangesOnApi
-from pathlib import Path
+from split_release_note import LibraryModule, create_changelog_entry, group_changes_by_api, ChangesOnApi, detect_modules
 
 dummy_module = LibraryModule(
     Path('release-note-generation/test/java-analyics-admin'),
@@ -81,6 +83,31 @@ def test_group_changes_by_api(self):
                                      ['No change']),
                          ['No change'])
 
+    def test_detect_modules_fallback(self):
+        with tempfile.TemporaryDirectory() as tmpdirname:
+            tmp_path = Path(tmpdirname)
+            module_path = tmp_path / "java-spanner"
+            module_path.mkdir()
+            
+            # Create minimal pom.xml
+            pom_path = module_path / "pom.xml"
+            with open(pom_path, "w") as f:
+                f.write('<project xmlns:mvn="http://maven.apache.org/POM/4.0.0"><mvn:version>1.0.0</mvn:version></project>')
+                
+            # Create .repo-metadata.json with api_shortname
+            metadata_path = module_path / ".repo-metadata.json"
+            with open(metadata_path, "w") as f:
+                json.dump({"api_shortname": "spanner"}, f)
+                
+            # Create CHANGELOG.md
+            changelog_path = module_path / "CHANGELOG.md"
+            changelog_path.touch()
+            
+            modules = detect_modules(tmp_path)
+            self.assertEqual(len(modules), 1)
+            self.assertEqual(modules[0].api_name, "spanner")
+            self.assertEqual(modules[0].version, "1.0.0")
+
 
 if __name__ == "__main__":
     unittest.main()

.github/workflows/ci.yaml

@@ -31,15 +31,14 @@ jobs:
       ci: ${{ steps.filter.outputs.ci }}
     steps:
     - uses: actions/checkout@v4
-    - uses: dorny/paths-filter@v3
+    - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
       id: filter
       with:
-        # we want to run tests if source code is changed or the scripts
-        # used to run the unit tests
         filters: |
           src:
-            - '**/*.java'
-            - '**/pom.xml'
+            - '!(java-bigquery|java-bigquerystorage|java-datastore|java-logging-logback|java-logging|java-spanner|java-storage|google-auth-library-java)/**/*.java'
+            - '!(java-bigquery|java-bigquerystorage|java-datastore|java-logging-logback|java-logging|java-spanner|java-storage|google-auth-library-java)/**/pom.xml'
+            - 'pom.xml'
           ci:
             - '.github/workflows/ci.yaml'
             - '.kokoro/**'
@@ -113,53 +112,62 @@ jobs:
     outputs:
       packages: ${{ steps.filter.outputs.changes }}
     steps:
-    - uses: dorny/paths-filter@v4
+    - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
       id: filter
       with:
         # For each library, run CI in split repos where there are changes in:
         # 1. Changes inside the split repo's module
         # 2. Java code changes in upstream modules: Auth Library and Sdk-Platform-Java
         # 3. Upstream dependency version changes: Shared-Deps and Gapic-Generator-Pom-Parent
         filters: |
+          google-auth-library-java:
+            - 'google-auth-library-java/**'
           java-bigquery:
             - 'java-bigquery/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
           java-bigquerystorage:
             - 'java-bigquerystorage/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
           java-datastore:
             - 'java-datastore/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
           java-logging-logback:
             - 'java-logging-logback/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
           java-logging:
             - 'java-logging/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
           java-spanner:
             - 'java-spanner/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
           java-storage:
             - 'java-storage/**'
             - 'google-auth-library-java/**/*.java'
+            - 'google-auth-library-java/**/pom.xml'
             - 'sdk-platform-java/**/*.java'
             - 'sdk-platform-java/java-shared-dependencies/**/pom.xml'
             - 'sdk-platform-java/gapic-generator-java-pom-parent/pom.xml'
@@ -236,6 +244,49 @@ jobs:
         JOB_TYPE: test
         JOB_NAME: units-8-runtime-${{matrix.java}}
       working-directory: ${{matrix.package}}
+  split-clirr:
+    runs-on: ubuntu-latest
+    needs: changes
+    strategy:
+      matrix:
+        package: ${{ fromJSON(needs.changes.outputs.packages) }}
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-java@v4
+      with:
+        distribution: temurin
+        java-version: 11
+    - run: .kokoro/build.sh
+      env:
+        BUILD_SUBDIR: ${{matrix.package}}
+        JOB_TYPE: clirr
+        JOB_NAME: clirr-${{matrix.package}}
+  split-dependencies:
+    runs-on: ubuntu-latest
+    needs: changes
+    strategy:
+      matrix:
+        package: ${{ fromJSON(needs.changes.outputs.packages) }}
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-java@v4
+      with:
+        distribution: temurin
+        java-version: 17
+    - run: .kokoro/dependencies.sh
+      env:
+        BUILD_SUBDIR: ${{matrix.package}}
+  required:
+    needs: [ changes, split-units, split-clirr, split-dependencies ]
+    name: conditional-required-check
+    if: ${{ always() }} # Always run even if any "needs" jobs fail
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Fail if any previous failure
+        if: ${{ needs.changes.outputs.packages != '[]' && contains(needs.*.result, 'failure') }}
+        run: exit 1
+      - name: Success otherwise
+        run: echo "Success!"
   windows:
     runs-on: windows-latest
     steps:
@@ -311,41 +362,14 @@ jobs:
     - name: validate generation configuration
       shell: bash
       run: |
-        docker run \
-          --rm \
+        bash generation/run_generator_docker.sh "${library_generation_image_tag}" "${{ github.base_ref || 'main' }}" \
+          -e GENERATOR_VERSION="${library_generation_image_tag}" \
           --quiet \
           -u "$(id -u):$(id -g)" \
           -v "$(pwd):${workspace_name}" \
           --entrypoint python \
-          gcr.io/cloud-devrel-public-resources/java-library-generation:"${library_generation_image_tag}" \
+          -- \
           /src/library_generation/cli/entry_point.py validate-generation-config
       env:
         library_generation_image_tag: 2.68.0
         workspace_name: /workspace
-
-# TODO: Uncomment the needed Github Actions
-#  dependencies:
-#    runs-on: ubuntu-latest
-#    strategy:
-#      matrix:
-#        java: [8, 11, 17]
-#    steps:
-#    - uses: actions/checkout@v3
-#    - uses: actions/setup-java@v3
-#      with:
-#        distribution: zulu
-#        java-version: ${{matrix.java}}
-#    - run: java -version
-#    - run: .kokoro/dependencies.sh
-#  clirr:
-#    runs-on: ubuntu-latest
-#    steps:
-#    - uses: actions/checkout@v3
-#    - uses: actions/setup-java@v3
-#      with:
-#        distribution: zulu
-#        java-version: 8
-#    - run: java -version
-#    - run: .kokoro/build.sh
-#      env:
-#        JOB_TYPE: clirr

…-java-create_additional_release_tag.yaml …flows/create_additional_release_tag.yaml.github/workflows/sdk-platform-java-create_additional_release_tag.yaml renamed to .github/workflows/create_additional_release_tag.yaml .github/workflows/sdk-platform-java-create_additional_release_tag.yaml renamed to .github/workflows/create_additional_release_tag.yaml

@@ -1,28 +1,11 @@
-name: sdk-platform-java Create additional tags for each release
+name: Create additional tags for each release
 
 on:
   release:
     types: [published]
   workflow_dispatch:
-
-env:
-  BUILD_SUBDIR: sdk-platform-java
 jobs:
-  filter:
-    runs-on: ubuntu-latest
-    outputs:
-      library: ${{ steps.filter.outputs.library }}
-    steps:
-    - uses: actions/checkout@v4
-    - uses: dorny/paths-filter@v3
-      id: filter
-      with:
-        filters: |
-          library:
-            - 'sdk-platform-java/**'
   build:
-    needs: filter
-    if: ${{ needs.filter.outputs.library == 'true' }}
     runs-on: ubuntu-latest
     permissions:
       # Permission to create tag
@@ -32,7 +15,7 @@ jobs:
     - name: Checkout code
       uses: actions/checkout@v4
       with:
-        token: ${{ secrets.GITHUB_TOKEN }}
+        token: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }}
     - name: Set up Git
       run: |
         git config --local user.email "action@github.com"