Skip to content

Commit 9d7bbd0

Browse files
authored
docs(deploy): document --property-graph schema-derived deploy mode (issue #286, PR 5) (#292)
Docs closeout for the scheduled-deploy parity. The bash deploy and Terraform both now support schema-derived mode (#289-#291); document it as the simple rename-free production path, with the explicit ontology/binding (migration-v5 compiled) path as the advanced override. - Deployment README: new 'Schema-derived deploy (--property-graph)' section after the deploy command -- one-artifact flow (point --property-graph at a placeholdered property_graph.sql with a sibling table_ddl.sql), the placeholder + no-compiled-only requirements, the read-only-events / writable-graph split, and an advanced note for explicit ontology+binding. - Terraform README: property_graph = true subsection (build with build_image.sh --property-graph; sets BQAA_PROPERTY_GRAPH; plan-time precondition vs compiled-only), plus tfvars override, the env-var list, and the build-image note. Docs only; no code change. This is the last PR for #286.
1 parent 781fd71 commit 9d7bbd0

2 files changed

Lines changed: 56 additions & 2 deletions

File tree

examples/migration_v5/periodic_materialization/README.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,44 @@ the logs, so you find out *now* whether the deploy actually
272272
works — not when the first scheduled fire happens six hours
273273
later.
274274

275+
### Schema-derived deploy (`--property-graph`)
276+
277+
The deploy above bundles the MAKO `ontology.yaml` + `binding.yaml`. For a
278+
**rename-free, codelab-style graph** you can skip those entirely and deploy
279+
from just a property graph — the same one-artifact flow the
280+
[codelab](../../../docs/codelabs/periodic_materialization.md) uses locally:
281+
282+
```bash
283+
./examples/migration_v5/periodic_materialization/deploy_cloud_run_job.sh \
284+
--project your-project \
285+
--region us-central1 \
286+
--events-dataset your_events_dataset \
287+
--graph-dataset your_graph_dataset \
288+
--schedule "0 */6 * * *" \
289+
--property-graph path/to/property_graph.sql
290+
```
291+
292+
`--property-graph` points at a `CREATE PROPERTY GRAPH` `.sql`; a placeholdered
293+
(`${PROJECT_ID}` / `${DATASET}`) `table_ddl.sql` must sit **next to it**. The
294+
deploy stages both (no `ontology.yaml`/`binding.yaml`), sets
295+
`BQAA_PROPERTY_GRAPH=property_graph.sql`, and the runtime derives the ontology +
296+
binding from the property graph + your live table schemas at run time (#286).
297+
Events stay read-only in your events dataset; the graph tables, schema lookup,
298+
and the state table all target the graph dataset.
299+
300+
Requirements / limits:
301+
302+
* Both files must be **placeholdered** (`${PROJECT_ID}` / `${DATASET}`) — the
303+
deploy refuses hardcoded references so the job can never derive against the
304+
wrong dataset.
305+
* Not compatible with `--extraction-mode=compiled-only` (no reference
306+
extractors are staged in derived mode); the deploy rejects that combination.
307+
308+
**Advanced — explicit ontology + binding.** Omit `--property-graph` to keep the
309+
default MAKO path: descriptions for AI prompting, entity inheritance, derived
310+
properties, column renames, and the hand-authored compiled extractor. That is
311+
the right path for migration v5 and any graph that outgrows schema-derived mode.
312+
275313
The script:
276314

277315
1. **Pre-creates the graph dataset** (`bq mk`, idempotent) so

examples/migration_v5/periodic_materialization/terraform/README.md

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,15 @@ Resolves [#186](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-
1111
| Graph dataset | `google_bigquery_dataset` | §1 — `bq mk` (pre-create so the runtime SA never needs `bigquery.datasets.create`) |
1212
| Runtime SA + scheduler-caller SA | `google_service_account` ×2 | §2 — `_ensure_sa` (split by default; `single_sa = true` collapses to one) |
1313
| Project + dataset IAM grants | `google_project_iam_member`, `google_bigquery_dataset_iam_member` | §3 — `_retry_iam gcloud projects add-iam-policy-binding` and the dataset-level grant block. Terraform's dependency graph handles the IAM-propagation race declaratively (the `depends_on` on the Cloud Run Job resource ensures grants land before the first invocation) |
14-
| Cloud Run v2 Job | `google_cloud_run_v2_job` | §4 — `gcloud run jobs deploy`. Same env-var set the bash deploy wires (`BQAA_PROJECT_ID`, `BQAA_EVENTS_DATASET_ID`, `BQAA_GRAPH_DATASET_ID`, `BQAA_LOCATION`, `BQAA_LOOKBACK_HOURS`, `BQAA_OVERLAP_MINUTES`, `BQAA_EXTRACTION_MODE`, `BQAA_MAX_RETRIES`, conditionally `BQAA_MAX_SESSIONS`, `BQAA_MAX_SESSION_AGE_HOURS`, `BQAA_REFERENCE_EXTRACTORS_MODULE`) |
14+
| Cloud Run v2 Job | `google_cloud_run_v2_job` | §4 — `gcloud run jobs deploy`. Same env-var set the bash deploy wires (`BQAA_PROJECT_ID`, `BQAA_EVENTS_DATASET_ID`, `BQAA_GRAPH_DATASET_ID`, `BQAA_LOCATION`, `BQAA_LOOKBACK_HOURS`, `BQAA_OVERLAP_MINUTES`, `BQAA_EXTRACTION_MODE`, `BQAA_MAX_RETRIES`, conditionally `BQAA_MAX_SESSIONS`, `BQAA_MAX_SESSION_AGE_HOURS`, `BQAA_REFERENCE_EXTRACTORS_MODULE`, and `BQAA_PROPERTY_GRAPH` when `property_graph = true`) |
1515
| `roles/run.invoker` on the job → scheduler SA | `google_cloud_run_v2_job_iam_member` | §5 — `_retry_iam gcloud run jobs add-iam-policy-binding` |
1616
| Cloud Scheduler trigger | `google_cloud_scheduler_job` | §6 — `gcloud scheduler jobs create http` with `--oauth-service-account-email` pointing at the scheduler SA |
1717

1818
## What's intentionally outside the module
1919

2020
**Container image build + publish.** The bash deploy builds the image from local sources via Cloud Buildpacks (`gcloud run jobs deploy --source $STAGING`), assembling a staging directory with `run_job.py` + `reference_extractor.py` + the demo artifacts + the vendored SDK source + `Procfile` + `requirements.txt` before invoking the build. That staging step is non-trivial — pointing `gcloud builds submit` directly at the `periodic_materialization/` directory would build an image that's missing the SDK source, the demo artifacts, and the Procfile.
2121

22-
The module takes the **published** image URI as the required `image_uri` variable. To produce a Terraform-compatible image, use the bundled [`../build_image.sh`](../build_image.sh) helper — it stages the exact layout `deploy_cloud_run_job.sh` produces in its temp dir, then runs `gcloud builds submit` against that staging dir. Same image contents either way; Terraform just consumes the publish artifact instead of doing the build inline.
22+
The module takes the **published** image URI as the required `image_uri` variable. To produce a Terraform-compatible image, use the bundled [`../build_image.sh`](../build_image.sh) helper — it stages the exact layout `deploy_cloud_run_job.sh` produces in its temp dir, then runs `gcloud builds submit` against that staging dir. Same image contents either way; Terraform just consumes the publish artifact instead of doing the build inline. For schema-derived deploys (`property_graph = true`), build the image with `build_image.sh --property-graph path/to/property_graph.sql` so the placeholdered `property_graph.sql` + `table_ddl.sql` are staged instead of `ontology.yaml`/`binding.yaml`.
2323

2424
```bash
2525
# From the repo root.
@@ -86,13 +86,29 @@ graph_dataset_id = "migration_v5_graph"
8686
schedule = "0 */6 * * *"
8787
image_uri = "us-central1-docker.pkg.dev/my-project/bqaa/periodic-materialization:v1"
8888
# Optional overrides — defaults match the bash deploy
89+
# property_graph = false # true → schema-derived mode (see below)
8990
# extraction_mode = "ai-fallback"
9091
# max_retries = 2
9192
# max_session_age_hours = 24
9293
# single_sa = false
9394
# location = "US"
9495
```
9596

97+
#### Schema-derived mode (`property_graph = true`)
98+
99+
For a **rename-free, codelab-style graph** you can skip the explicit
100+
`ontology.yaml` / `binding.yaml` and derive the spec from a property graph (the
101+
[#286](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/286)
102+
one-artifact flow). Set `property_graph = true` and build the image with
103+
`build_image.sh --property-graph path/to/property_graph.sql` (a placeholdered
104+
`${PROJECT_ID}` / `${DATASET}` `table_ddl.sql` must sit next to it). The module
105+
then sets `BQAA_PROPERTY_GRAPH=property_graph.sql` on the Job so the runtime
106+
derives the ontology + binding from the property graph + your live table
107+
schemas. `property_graph = true` is rejected at plan time with
108+
`extraction_mode = "compiled-only"` (no reference extractors are staged in
109+
derived mode). Leave `property_graph = false` (default) for the explicit MAKO /
110+
compiled-extractor path.
111+
96112
### 2. Init + plan + apply
97113

98114
```bash

0 commit comments

Comments
 (0)