Skip to content

Commit fb72bbf

Browse files
authored
Syncing data cloud skills (#41)
1 parent f3ccd0d commit fb72bbf

1 file changed

Lines changed: 41 additions & 30 deletions

File tree

skills/dataform-bigquery/SKILL.md

Lines changed: 41 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,23 @@ Follow these steps when fulfilling Dataform-related requests:
6060
* **If `workflow_settings.yaml` IS found**:
6161
* Run `dataform compile <PROJECT_DIR>` to compile the pipeline and get
6262
an overview of existing files and the DAG.
63+
- Once the repository is located or initialized, check if
64+
`.df-credentials.json` is present in the Dataform project directory. If
65+
absent, ask the user to run `dataform init-creds` to create the credentials
66+
file. If the user cannot initialize the credentials, write the
67+
`.df-credentials.json` file manually, following the format below. Replace
68+
`<PROJECT_ID>` with a Google Cloud project for billing (e.g., obtained via
69+
`gcloud config get-value project`) and `<LOCATION>` with the appropriate
70+
region (e.g., obtained via `gcloud config get compute/region` or defaulting
71+
to `us-central1` if unspecified).
72+
73+
```json
74+
{
75+
"projectId": "<PROJECT_ID>",
76+
"location": "<LOCATION>"
77+
}
78+
```
79+
6380
- Use the compiled graph as the **source of truth** for existing assets.
6481

6582
### 2. Gather Information
@@ -119,28 +136,23 @@ For non-trivial requests, create a clear specification before implementation:
119136
### 6. Validate & Compile
120137

121138
- Run `dataform compile` to catch syntax and dependency errors.
122-
- Check if `.df-credentials.json` is present in the Dataform project
123-
directory.
124-
125-
- **If present:** Run `dataform run --dry-run` for validation.
126-
- **If absent:** Ask the user to run `dataform init-creds` to create the
127-
credentials file.
128-
- **If credentials cannot be initialized:** Fall back to using `dataform
129-
compile`, manual SQL inspection, and `bq query --dry_run` for
130-
validation.
131-
132-
> [!IMPORTANT]
133-
>
134-
> If `dataform run --dry-run` fails, inspect the error message. If the
135-
> failure is ONLY due to "Table not found" errors for nodes defined
136-
> within the current Dataform project (which occurs when upstream
137-
> dependencies haven't been materialized in BigQuery), then this
138-
> specific error may be ignored. If the dry run fails for ANY other
139-
> reason (such as SQL syntax errors, permission errors, or references to
140-
> tables not defined in the project), these errors MUST be addressed. If
141-
> only "Not found" errors for unmaterialized project tables are present,
142-
> rely on `dataform compile`, manual SQL inspection, and `bq query
143-
> --dry_run` for verification.
139+
- If `.df-credentials.json` is successfully set up (from Step 1), run
140+
`dataform run --dry-run` for validation.
141+
- If `.df-credentials.json` could not be initialized, fall back to using
142+
`dataform compile`, manual SQL inspection, and `bq query --dry_run` for
143+
validation.
144+
145+
> [!IMPORTANT]
146+
>
147+
> If `dataform run --dry-run` fails, inspect the error message. If the
148+
> failure is ONLY due to "Table not found" errors for nodes defined within
149+
> the current Dataform project (which occurs when upstream dependencies
150+
> haven't been materialized in BigQuery), then this specific error may be
151+
> ignored. If the dry run fails for ANY other reason (such as SQL syntax
152+
> errors, permission errors, or references to tables not defined in the
153+
> project), these errors MUST be addressed. If only "Not found" errors for
154+
> unmaterialized project tables are present, rely on `dataform compile`,
155+
> manual SQL inspection, and `bq query --dry_run` for verification.
144156

145157
- Validate SQL logic of changed nodes and fix any errors.
146158

@@ -153,20 +165,19 @@ For non-trivial requests, create a clear specification before implementation:
153165

154166
- Repeat steps 5–6 until the request is fully satisfied.
155167

156-
## Credentials for `dataform run`
168+
## Credentials for `dataform run` and `dataform run --dry-run`
157169

158170
The command `dataform run` executes your Dataform pipeline in BigQuery but
159171
requires credentials to be set up in a `.df-credentials.json` file in your
160172
project directory.
161173

162-
Generate pipeline code and ensure it compiles via `dataform compile`. Use
163-
`dataform run --dry-run` for validation only if the `.df-credentials.json` file
164-
is present. MUST NOT execute a real `dataform run` without explicit user
165-
request.
174+
Generate pipeline code and ensure it compiles via `dataform compile`. Validate
175+
the pipeline using `dataform run --dry-run` once the `.df-credentials.json` file
176+
is successfully created (as instructed in the Understand the Current State
177+
step). MUST NOT execute a real `dataform run` without explicit user request.
166178

167-
If `.df-credentials.json` is missing, instruct the user to run `dataform
168-
init-creds` in their terminal. If they are unable to get the credentials
169-
initialized, fall back on other methods of validation, such as `dataform
179+
If `.df-credentials.json` could not be initialized via `dataform init-creds` or
180+
manual creation, fall back on other methods of validation, such as `dataform
170181
compile`, manual SQL inspection, and `bq query --dry_run`.
171182

172183
## Incremental / Append Operations

0 commit comments

Comments
 (0)