feat: update skills from cloudtop

Author: belluru

skills/bigquery-data-transfer-service/SKILL.md

@@ -37,6 +37,7 @@ Before generating configurations, discover the actual values for the target
 project and region.
 
 > [!TIP]
+>
 > If `deployment.yaml` already exists in the repository root, prioritize
 > extracting `project` and `region` from the target environment configuration
 > (e.g., `dev`).
@@ -45,6 +46,7 @@ project and region.
 2.  **Region**: `gcloud config get compute/region`
 
 > [!TIP]
+>
 > Use these commands to replace placeholders like `<PROJECT_ID>` with actual
 > values. Always remove associated comments that start with TODO once replaced.
 
@@ -68,8 +70,8 @@ region.
         -   Check if the transfer has at least one successful run: `bq ls
             --transfer_run --transfer_config=<RESOURCE_NAME>`
         -   If found: Use existing transfer config.
-        -   If not found: Confirm with user if it's ok to trigger
-            the transfer run.
+        -   If not found: Confirm with user if it's ok to trigger the transfer
+            run.
 
     -   **Multiple Transfers Found**:
 
@@ -80,8 +82,8 @@ region.
 
         -   Ask user if they want to enable it or create a new one.
         -   To Enable: Instruct the user to update the transfer configuration
-            within their `deployment.yaml` file by setting the `disabled`
-            field to `false` for the specific transfer resource.
+            within their `deployment.yaml` file by setting the `disabled` field
+            to `false` for the specific transfer resource.
 
     -   **No Transfers Found**: Proceed to create new if needed.
 
@@ -90,10 +92,12 @@ region.
 If creating a new transfer, discover the required parameters using the REST API
 and validate them with the user.
 
-> [!TIP] If `<DATA_SOURCE_ID>` is unknown, run the discovery script
-> without `<DATA_SOURCE_ID>` argument to list available source IDs
-> (e.g., `google_cloud_storage`).
-> It uses the derived project and location from Step 0.
+> [!TIP]
+>
+> If `<DATA_SOURCE_ID>` is unknown, run the discovery script without
+> `<DATA_SOURCE_ID>` argument to list available source IDs (e.g.,
+> `google_cloud_storage`). It uses the derived project and location from Step 0.
+>
 > ```bash
 > python3 scripts/bigquery_dts.py --project_id=<PROJECT_ID>
 > ```
@@ -106,24 +110,28 @@ and validate them with the user.
     python3 scripts/bigquery_dts.py --project_id=<PROJECT_ID> <DATA_SOURCE_ID> <REGION>
     ```
 
-    > [!IMPORTANT] Run this command every time a new transfer is being planned.
+    > [!IMPORTANT]
+    >
+    > Run this command every time a new transfer is being planned.
 
-2.  > [!CAUTION] **Mandatory User Questionnaire (CRITICAL)**:
+2.  > [!CAUTION]
+    >
+    > **Mandatory User Questionnaire (CRITICAL)**:
 
     -   **Explicitly identify ALL specific parameters** returned by the
         discovery script. **You MUST NOT generalize or vaguely summarize them.**
     -   **OAuth Authorization (Google Data Sources)**: For Google ecosystem data
         sources (Google Ads, Youtube, etc.), if the user is not using a service
         account to configure the DTS transfer config (meaning the user is using
         End User Credentials or EUC to configure the transfer config), then
-        generate an OAuth URI. Ask the user to visit this URL to authorize.
-        Once the user provides the versionInfo code, use the code as
+        generate an OAuth URI. Ask the user to visit this URL to authorize. Once
+        the user provides the versionInfo code, use the code as
         `definition.versionInfo` in `deployment.yaml` and then you can proceed.
-    -   If any parameters are related to authentication,
-        explicitly ask the user to provide the Secret Manager Resource ID
-        (e.g., projects/my-project/secrets/my-secret) for these parameters
-    -   Present every required parameter to the user BEFORE generating
-        config files.
+    -   If any parameters are related to authentication, explicitly ask the user
+        to provide the Secret Manager Resource ID (e.g.,
+        projects/my-project/secrets/my-secret) for these parameters
+    -   Present every required parameter to the user BEFORE generating config
+        files.
     -   Ask for verification of assets/tables to be ingested.
 
 3.  **Wait for User Response**: You **MUST NOT** proceed until parameters are

skills/dataform-bigquery/SKILL.md

@@ -7,7 +7,7 @@ description: Expertise in generating clean, correct, and efficient Dataform pipe
   up a new Dataform project or configuring workflow_settings.yaml.
 license: Apache-2.0
 metadata:
-  version: v1
+  version: v2
   publisher: google
 ---
 
@@ -270,10 +270,8 @@ Usage in models:
 SELECT * FROM ${ref("my_iceberg_table")}
 ```
 
-> [!WARNING]
->
-> You cannot create a BigQuery view directly from a source BigLake table (using
-> 4-part naming). It needs to be a native BigQuery table.
+You cannot create a BigQuery view directly from a source BigLake table (using
+4-part naming). This feature is only for native BigQuery tables.
 
 ## Unit Testing
 

skills/dbt-bigquery/SKILL.md

@@ -7,7 +7,7 @@ description: Expert guidance for creating, modifying, and optimizing dbt pipelin
   **setting up a new dbt project** or configuring existing one
 license: Apache-2.0
 metadata:
-  version: v1
+  version: v2
   publisher: google
 ---
 
@@ -45,8 +45,7 @@ Follow these steps when fulfilling dbt-related requests:
 ### 1. Understand the Current State
 
 -   Locate the dbt project root by searching for a `dbt_project.yml` file.
-    -   **If `dbt_project.yml` is NOT found**: Assume the repository is
-        uninitialized and guide the user through `dbt init`.
+    -   **If `dbt_project.yml` is NOT found**: Assume the repository/project is uninitialized.
 -   Compile the dbt pipeline (`dbt compile`) to map the existing DAG.
 -   Use the compiled graph as the **source of truth** for existing assets.
 
@@ -116,8 +115,14 @@ Follow these steps when fulfilling dbt-related requests:
             dbt-bigquery`).
         -   Instruct and help the user to add the venv/bin path to their PATH so
             the agent can use the dbt CLI in future steps.
--   **Repo Initialization**: If the repository or dbt project does not exist,
-    instruct on how to initialize it.
+-   **Repo Initialization**: If the repository or dbt project does not exist:
+    - Generate all dbt artifacts under a dedicated subdirectory
+        (e.g., `dbt/`) rather than the root.
+    -   **Silent & Scaffolded Initialization**: Initialize silently.
+        Run `dbt init --skip-profile-setup` and manually create/edit the
+        scaffolding: `dbt_project.yml`, `profiles.yml`,
+        and other directories for `models/` and `tests/` as needed
+        (i.e: if dbt init fails).
 -   **Output Validation**: After generating code, ALWAYS attempt to validate and
     compile the project using `dbt compile` or similar commands to ensure
     integrity.
@@ -169,14 +174,16 @@ acceptable."
 
 ### Project & Profiles Config
 
--   When initializing a new dbt project ensure `dbt_project.yml` is created with
-    correct settings.
+-   Always generate the dbt project and files within a dedicated folder (e.g.,
+    `dbt/`) rather than the root folder to avoid orchestrator errors.
+-   When initializing a new dbt project ensure `dbt_project.yml` is created
+    with correct settings.
 -   **Profiles Config**: ALWAYS ensure that a `profiles.yml` file is generated
-    inside the dbt project folder alongside `dbt_project.yml` (or explicitly
-    point `DBT_PROFILES_DIR` to it). Uncreated profiles are a leading cause of
-    DAG pipeline failures (e.g., "Could not find profile named 'X'"). The
-    `profiles.yml` must match the profile requested in `dbt_project.yml` and map
-    correct BigQuery settings (project, dataset, location).
+    inside the dedicated dbt project folder alongside `dbt_project.yml` (or
+    explicitly point `DBT_PROFILES_DIR` to it). Uncreated profiles are a leading
+    cause of DAG pipeline failures (e.g., "Could not find profile named 'X'").
+    The `profiles.yml` must match the profile requested in `dbt_project.yml` and
+    map correct BigQuery settings (project, dataset, location).
 
 ### Model Configuration
 
@@ -210,11 +217,9 @@ The `dbt-bigquery` adapter does not natively support 4-part
 If you don't use environment prefixes for schemas, you can concatenate the
 `catalog` and `namespace` (dataset) into the `schema` field.
 
-> [!WARNING]
->
-> This approach breaks standard dbt environment management (e.g.,
-> `generate_schema_name`) if it attempts to prefix the combined string (e.g.,
-> `dev_my_catalog.my_namespace` is invalid in BigQuery).
+This approach is incompatible with standard dbt environment management (e.g.,
+ `generate_schema_name`) if it attempts to prefix the combined string (e.g.,
+ `dev_my_catalog.my_namespace` is invalid in BigQuery).
 
 ```yaml
 version: 2

skills/discovering-gcp-data-assets/SKILL.md

@@ -19,32 +19,37 @@ description: |
     - Assets are outside Google Cloud
 license: Apache-2.0
 metadata:
-  version: v1
+  version: v4
   publisher: google
 ---
 
 # Instructions
 
-## Step 1: Handle Public Datasets or Proceed to Search
+## Step 1: Prioritize Assets from the Conversation
 
-Dataplex Entries Lookup provides the richest metadata for data assets. You MUST
+If the asset was created or mentioned earlier in the same conversation, then
+proceed with that asset instead of searching. Skip steps 2, 3, and 4.
+
+## Step 2: Handle Public Datasets or Proceed to Search
+
+Dataplex Lookup Context provides the richest metadata for data assets. You MUST
 prioritize using it for all Google Cloud assets, even if you already know their
 IDs.
 
 -   **Public Datasets (Direct Inspection)**: If the requested asset belongs to
-    the `bigquery-public-data` project, Dataplex Entries Lookup will fail. You
-    MUST skip Steps 2 and 3 and inspect the table directly using the `bq` CLI or
+    the `bigquery-public-data` project, Dataplex Lookup Context will fail. You
+    MUST skip Steps 3 and 4 and inspect the table directly using the `bq` CLI or
     BigQuery MCP tools instead.
--   **All Other Assets (Proceed to Step 2)**: For all other BigQuery, Cloud
+-   **All Other Assets (Proceed to Step 3)**: For all other BigQuery, Cloud
     Storage, Spanner, BigLake Iceberg or general GCP data assets (whether their
-    IDs are known or missing), you MUST proceed to **Step 2** to search the
+    IDs are known or missing), you MUST proceed to **Step 3** to search the
     Dataplex catalog and obtain their full Entry Name.
 
-## Step 2: Execute Discovery Search
+## Step 3: Execute Discovery Search
 
 You MUST use the Dataplex search command to discover assets and retrieve their
 full `projects/...` entry names. This step is required even if you already know
-the asset's short ID (e.g., `my_dataset.my_table`), because Step 3 strictly
+the asset's short ID (e.g., `my_dataset.my_table`), because Step 4 strictly
 requires the full entry name.
 
 > [!IMPORTANT] The `--project` parameter MUST ALWAYS be provided. This
@@ -86,8 +91,9 @@ Use this for exact keyword matches or technical strings (e.g., `name:order_v2`).
     -   Use `:` for token/substring matches (e.g., `name:sales`).
     -   Use `=` for exact matches. REQUIRED for `system`, `type`, and
         `location`.
--   **Singular Keywords**: ALWAYS convert plurals to singular (e.g., "product"
-    NOT "products").
+-   **Singular Keywords**: When performing keyword search, ALWAYS convert
+    plurals to singular (e.g., "product" NOT "products"). Semantic search
+    handles singular/plural variations and synonyms automatically.
 -   **Scope Restriction**: You SHOULD restrict the search scope using a `parent`
     filter if the project or dataset is known (e.g.,
     `parent:projects/<PROJECT_ID>`).
@@ -126,26 +132,50 @@ gcloud dataplex entries search "<KEYWORD_SEARCH_QUERY>" \
   --limit=50
 ```
 
-*Criteria*: Once candidate assets are returned, proceed to Step 3 using the
+> [!IMPORTANT] Handling Search Results and Avoiding Loops:
+>
+> 1.  **No Results:** If the search returns no entries:
+>     *   **Variation Rule:** You may try AT MOST 3 variations of the search
+>         query (e.g., switching AND/OR clauses, adding/removing `parent:`,
+>         removing `projectid:` or `location:`, trying `fully_qualified_name=`).
+>     *   **Stop Rule:** If after 3 attempts no results are found, STOP and
+>         inform the user. Ask for clarification, specifically the Dataplex
+>         **full entry name** if known, or identifiers such as **project ID**,
+>         **dataset ID**, or **instance ID** to help narrow the search. Example:
+>         "I couldn't find any tables matching that description after several
+>         attempts. If you know the Dataplex full entry name (`projects/...`),
+>         please provide it. Otherwise, please provide any identifiers you know,
+>         such as project, dataset, or instance name, to help locate the asset."
+> 2.  **Multiple Results:**
+>     *   If more than 10 results are returned, state that many matches were
+>         found. Show the names of the first 5 entries and ask for
+>         clarification.
+>     *   If 2-10 results are returned and you cannot definitively choose, list
+>         them and ask the user.
+> 3.  **Single Result:** Proceed to Step 3 with the full entry name.
+> 4.  **Avoid Infinite Loops:** MUST NOT re-run identical or near-identical
+>     queries. If Dataplex fails to return the expected asset, prioritize asking
+>     the user for the exact resource ID or using Fallback 2 (Native Tools).
+
+*Criteria*: Once candidate assets are returned, proceed to Step 4 using the
 **full entry names** from the search results.
+## Step 4: Lookup Context
 
-## Step 3: Lookup Entry
-
-You MUST use the **Entries Lookup** command to fetch schema and deep metadata
-for the relevant results obtained from Step 2.
+You MUST use the **Lookup Context** command to fetch schema and deep metadata
+for the relevant results obtained from Step 3.
 
-> [!IMPORTANT] The argument MUST be the **name** (starting with `projects/`)
-> returned by the search result. Passing short table IDs, GCS URIs, or fully
-> qualified `bigquery:` prefixes is PROHIBITED and will fail.
+> [!IMPORTANT] The `--resources` parameter MUST be the **full name** (starting
+> with `projects/`) returned by the search result. Passing short table IDs, GCS
+> URIs, or fully qualified `bigquery:` prefixes is PROHIBITED and will fail.
 
 ### Command Execution
 
-Use the `lookup_entry` MCP tool
+Use the `lookup_context` MCP tool
 
 OR
 
 ```bash
-gcloud dataplex entries lookup "<FULL_ENTRY_NAME>"
+gcloud dataplex context lookup --resources="<FULL_ENTRY_NAME>"
 ```
 
 *Completion Criteria*: The command returns the detailed schema and business
@@ -155,7 +185,7 @@ context.
 
 ## Troubleshooting
 
-### Lookup Fails or "Resource not found"
+### Context Lookup Fails or "Resource not found"
 
 -   **Cause**: Short table names were used improperly.
 -   **Fix**: Ensure you use the correct entry name format from the search
@@ -167,12 +197,24 @@ context.
 -   **Fix**: Switch to singular keywords. For semantic search, try more
     descriptive natural language.
 
-### Lookup Fails with "NOT_FOUND" (despite correct format)
+### Context Lookup Fails with "NOT_FOUND" (despite correct format)
 
 -   **Cause**: The table belongs to a project (e.g., `bigquery-public-data`)
     that has not fully synchronized its metadata with the Dataplex Universal
-    Catalog. While the entry appears in search, `entries lookup` is unavailable.
+    Catalog. While the entry appears in search, `context lookup` is unavailable.
 -   **Fix**: Fall back to direct inspection using native tools (e.g., `bq` CLI).
+-   **Stop Rule:** If the native tool (e.g., `bq show`) also returns "Not
+    Found", STOP. Do not restart the Dataplex discovery loop. Specifically ask
+    the user to verify the **project ID** and **table ID**.
+
+### Breaking the Research Loop
+
+If you find yourself repeatedly searching for the same asset:
+
+1.  **STOP.**
+2.  State what you have tried (e.g., "I tried searching Dataplex with X and Y,
+    and checked `bq show`").
+3.  Ask the user for the exact project, dataset, and table ID.
 
 ### Search Fails with "--project: Must be specified."