feat(deploy): --property-graph mode for the Cloud Run deploy + image scripts (issue #286, PR 3) (#290)

caohy1988 · web-flow · commit 9127ba1b3af6 · 2026-06-05T16:31:38.000-07:00
* feat(deploy): --property-graph mode for the Cloud Run deploy + image scripts (issue #286, PR 3) Adds schema-derived mode to deploy_cloud_run_job.sh and build_image.sh, on the #288/#289 runtime. One artifact contract: --property-graph PATH points at a CREATE PROPERTY GRAPH .sql, and table_ddl.sql is staged from the same directory (codelab bundles ship both together). deploy_cloud_run_job.sh: - New --property-graph PATH. Validates at the deploy boundary (before any gcloud): rejects --extraction-mode=compiled-only (no reference extractors are staged in derived mode -> would empty_extract), and requires a sibling table_ddl.sql (derive mode reads INFORMATION_SCHEMA, so the tables must be bootstrappable). Stages only property_graph.sql + table_ddl.sql (no ontology/binding/reference_extractor), and sets BQAA_PROPERTY_GRAPH= property_graph.sql so the runtime derives the spec. Explicit ontology+binding (migration-v5 / compiled-extractor) path unchanged when --property-graph is omitted. build_image.sh: mirror --property-graph staging (graph + sibling table DDL vs the explicit ontology/binding/extractor set). Tests: shell tests invoke the deploy script and assert the two rejection paths (compiled-only, missing sibling table_ddl) exit non-zero with actionable messages before any gcloud; static assertions pin the staging + BQAA_PROPERTY_GRAPH contract for both scripts. 5 tests pass; bash -n clean. Next (PR 4): Terraform property-graph mode; (PR 5) deploy docs. * fix(deploy): enforce the placeholder contract in property-graph mode (review P1) The scripts validated that property_graph.sql + sibling table_ddl.sql exist, but not that they are placeholdered. That left the original #286 failure mode open: a hardcoded graph DDL (e.g. the migration-v5 snapshot pointing at the canonical demo dataset) could be staged and derive against the wrong dataset. Both deploy_cloud_run_job.sh and build_image.sh now reject property-graph mode unless BOTH artifacts contain ${PROJECT_ID} and ${DATASET} (grep -qF), with an actionable error pointing at the offending file. Caught at the deploy/build boundary, before any gcloud or image bake -- so Terraform-built images can't bake wrong artifacts either. Tests: hardcoded property_graph.sql and hardcoded table_ddl.sql each rejected with the placeholder message; static assertions pin the grep checks in both scripts. 7 tests pass; bash -n clean.
diff --git a/examples/migration_v5/periodic_materialization/build_image.sh b/examples/migration_v5/periodic_materialization/build_image.sh
@@ -56,6 +56,11 @@ Optional:
   --tag TAG              Image tag (default: short git SHA of HEAD, or
                          a timestamp if HEAD isn't a git ref).
   --create-repo          Create the AR repo if it doesn't exist.
+  --property-graph PATH   Schema-derived mode (#286): stage this CREATE
+                         PROPERTY GRAPH .sql plus a sibling table_ddl.sql
+                         instead of ontology.yaml/binding.yaml/
+                         reference_extractor.py. The runtime derives the
+                         ontology + binding from them.
   -h | --help            Show this help.
 
 Outputs (last line of stdout):
@@ -67,16 +72,18 @@ EOF
 }
 
 CREATE_REPO=false
+PROPERTY_GRAPH=""
 while [[ $# -gt 0 ]]; do
   case "$1" in
-    --project)      PROJECT="$2"; shift 2 ;;
-    --repo)         REPO="$2"; shift 2 ;;
-    --region)       REGION="$2"; shift 2 ;;
-    --image)        IMAGE="$2"; shift 2 ;;
-    --tag)          TAG="$2"; shift 2 ;;
-    --create-repo)  CREATE_REPO=true; shift ;;
-    -h|--help)      usage 0 ;;
-    *)              echo "Unknown arg: $1" >&2; usage 1 ;;
+    --project)        PROJECT="$2"; shift 2 ;;
+    --repo)           REPO="$2"; shift 2 ;;
+    --region)         REGION="$2"; shift 2 ;;
+    --image)          IMAGE="$2"; shift 2 ;;
+    --tag)            TAG="$2"; shift 2 ;;
+    --create-repo)    CREATE_REPO=true; shift ;;
+    --property-graph) PROPERTY_GRAPH="$2"; shift 2 ;;
+    -h|--help)        usage 0 ;;
+    *)                echo "Unknown arg: $1" >&2; usage 1 ;;
   esac
 done
 
@@ -85,6 +92,29 @@ if [[ -z "$PROJECT" || -z "$REPO" ]]; then
   usage 1
 fi
 
+TABLE_DDL_SRC=""
+if [[ -n "$PROPERTY_GRAPH" ]]; then
+  if [[ ! -f "$PROPERTY_GRAPH" ]]; then
+    echo "Error: --property-graph file not found: $PROPERTY_GRAPH" >&2
+    usage 1
+  fi
+  TABLE_DDL_SRC="$(dirname "$PROPERTY_GRAPH")/table_ddl.sql"
+  if [[ ! -f "$TABLE_DDL_SRC" ]]; then
+    echo "Error: schema-derived mode needs a 'table_ddl.sql' next to the property graph; not found: $TABLE_DDL_SRC" >&2
+    usage 1
+  fi
+  # Placeholder contract (#286): refuse to bake a hardcoded graph DDL into the
+  # image. Both artifacts must use \${PROJECT_ID} / \${DATASET} so the runtime
+  # retargets them to the customer's project + graph dataset.
+  for _pg_artifact in "$PROPERTY_GRAPH" "$TABLE_DDL_SRC"; do
+    if ! grep -qF '${PROJECT_ID}' "$_pg_artifact" \
+      || ! grep -qF '${DATASET}' "$_pg_artifact"; then
+      echo "Error: schema-derived mode requires placeholdered artifacts: $_pg_artifact must contain \${PROJECT_ID} and \${DATASET}. Hardcoded graph DDL would derive against the wrong dataset." >&2
+      usage 1
+    fi
+  done
+fi
+
 if [[ -z "$TAG" ]]; then
   # Default tag: short git SHA if HEAD is reachable, timestamp
   # otherwise. Either keeps tags monotonically increasing so a
@@ -131,10 +161,17 @@ echo "==> staging build context at $STAGING" >&2
 # pins these copies to lock the contract.
 
 cp "${SCRIPT_DIR}/run_job.py" "$STAGING/"
-cp "${ARTIFACTS_DIR}/ontology.yaml" "$STAGING/"
-cp "${ARTIFACTS_DIR}/binding.yaml" "$STAGING/"
-cp "${ARTIFACTS_DIR}/table_ddl.sql" "$STAGING/"
-cp "${ARTIFACTS_DIR}/reference_extractor.py" "$STAGING/"
+if [[ -n "$PROPERTY_GRAPH" ]]; then
+  # Schema-derived mode: stage the property graph + its companion table DDL.
+  cp "$PROPERTY_GRAPH" "$STAGING/property_graph.sql"
+  cp "$TABLE_DDL_SRC" "$STAGING/table_ddl.sql"
+else
+  # Explicit ontology + binding (migration-v5 / compiled-extractor path).
+  cp "${ARTIFACTS_DIR}/ontology.yaml" "$STAGING/"
+  cp "${ARTIFACTS_DIR}/binding.yaml" "$STAGING/"
+  cp "${ARTIFACTS_DIR}/table_ddl.sql" "$STAGING/"
+  cp "${ARTIFACTS_DIR}/reference_extractor.py" "$STAGING/"
+fi
 
 mkdir -p "$STAGING/sdk_src/src"
 cp -r "$REPO_ROOT/src/bigquery_agent_analytics" "$STAGING/sdk_src/src/"
diff --git a/examples/migration_v5/periodic_materialization/deploy_cloud_run_job.sh b/examples/migration_v5/periodic_materialization/deploy_cloud_run_job.sh
@@ -101,6 +101,7 @@ MAX_SESSIONS=""
 JOB_NAME="bqaa-periodic-materialization"
 SMOKE=false
 EXTRACTION_MODE="ai-fallback"
+PROPERTY_GRAPH=""
 MAX_SESSION_AGE_HOURS=""
 # Production posture by default: split runtime + scheduler-caller
 # SAs (issue #182). ``--single-sa`` is the escape hatch for
@@ -148,6 +149,14 @@ Optional:
                                roles/aiplatform.user grant and idempotently
                                removes any pre-existing grant from a prior
                                ai-fallback deploy of the same SA.
+  --property-graph PATH        Schema-derived mode (#286): derive the
+                               ontology + binding from this CREATE PROPERTY
+                               GRAPH .sql file plus the table schemas, instead
+                               of staging ontology.yaml/binding.yaml. A
+                               placeholdered (\${PROJECT_ID}/\${DATASET})
+                               table_ddl.sql must sit next to it. Use for
+                               rename-free graphs; not compatible with
+                               --extraction-mode=compiled-only.
   --max-session-age-hours N    Enable the orphan-session watchdog (issue
                                #180). When set, each cron pass additionally
                                scans for sessions whose first event is
@@ -214,6 +223,7 @@ while [[ $# -gt 0 ]]; do
     --job-name)        require_arg "$1" "${2-}"; JOB_NAME="$2"; shift 2 ;;
     --smoke)           SMOKE=true; shift ;;
     --extraction-mode) require_arg "$1" "${2-}"; EXTRACTION_MODE="$2"; shift 2 ;;
+    --property-graph)  require_arg "$1" "${2-}"; PROPERTY_GRAPH="$2"; shift 2 ;;
     --max-session-age-hours)
                        require_arg "$1" "${2-}"; MAX_SESSION_AGE_HOURS="$2"; shift 2 ;;
     --single-sa)       SINGLE_SA=true; shift ;;
@@ -253,6 +263,45 @@ case "$EXTRACTION_MODE" in
     ;;
 esac
 
+# Spec input mode (#286). ``--property-graph`` selects schema-derived
+# mode: the ontology + binding are derived from the property graph + the
+# table schemas, so no ``ontology.yaml`` / ``binding.yaml`` is staged.
+# Unset = the explicit migration-v5 ontology+binding (compiled-extractor)
+# path. Exactly one mode is in effect.
+TABLE_DDL_SRC=""
+if [[ -n "$PROPERTY_GRAPH" ]]; then
+  # Derived mode has no reference extractors staged, so compiled-only
+  # would empty_extract at runtime — reject it at the deploy boundary.
+  if [[ "$EXTRACTION_MODE" == "compiled-only" ]]; then
+    echo "Error: --property-graph (schema-derived mode) does not support --extraction-mode=compiled-only: no reference extractors are staged in derived mode. Use 'ai-fallback', or deploy the explicit ontology/binding path." >&2
+    exit 1
+  fi
+  if [[ ! -f "$PROPERTY_GRAPH" ]]; then
+    echo "Error: --property-graph file not found: $PROPERTY_GRAPH" >&2
+    exit 1
+  fi
+  # The derived path also needs the graph tables to exist before the first
+  # run (it reads INFORMATION_SCHEMA), so a placeholdered table_ddl.sql must
+  # sit next to the property graph.
+  TABLE_DDL_SRC="$(dirname "$PROPERTY_GRAPH")/table_ddl.sql"
+  if [[ ! -f "$TABLE_DDL_SRC" ]]; then
+    echo "Error: schema-derived mode also needs a 'table_ddl.sql' next to the property graph (so the graph tables can be bootstrapped); not found: $TABLE_DDL_SRC" >&2
+    exit 1
+  fi
+  # Enforce the placeholder contract (#286). Both artifacts must use
+  # \${PROJECT_ID} / \${DATASET} so the runtime retargets them to the
+  # customer's project + graph dataset. A hardcoded graph DDL (e.g. the
+  # migration-v5 snapshot pointing at a canonical demo dataset) would derive
+  # against the wrong dataset -- reject it here, not after deploy.
+  for _pg_artifact in "$PROPERTY_GRAPH" "$TABLE_DDL_SRC"; do
+    if ! grep -qF '${PROJECT_ID}' "$_pg_artifact" \
+      || ! grep -qF '${DATASET}' "$_pg_artifact"; then
+      echo "Error: schema-derived mode requires placeholdered artifacts: $_pg_artifact must contain \${PROJECT_ID} and \${DATASET} so it can be retargeted to your project/graph dataset. Hardcoded graph DDL would derive against the wrong dataset. Use placeholdered, rename-free artifacts, or deploy the explicit --ontology/--binding path." >&2
+      exit 1
+    fi
+  done
+fi
+
 # Validate ``--max-retries`` at the boundary (issue #183). A typo
 # like ``--max-retries=-1`` would otherwise be forwarded to
 # ``gcloud run jobs deploy`` which rejects it with a less obvious
@@ -586,17 +635,26 @@ STAGING="$(mktemp -d -t bqaa-cloud-run-job-XXXXXXXX)"
 
 echo "==> staging at $STAGING"
 cp "${SCRIPT_DIR}/run_job.py" "$STAGING/"
-cp "${ARTIFACTS_DIR}/ontology.yaml" "$STAGING/"
-cp "${ARTIFACTS_DIR}/binding.yaml" "$STAGING/"
-cp "${ARTIFACTS_DIR}/table_ddl.sql" "$STAGING/"
-# Stage the reference extractor module next to ``run_job.py`` so
-# Python can import it via ``BQAA_REFERENCE_EXTRACTORS_MODULE=
-# reference_extractor`` (Buildpacks sets the container working
-# directory to the staging dir's contents, which puts the
-# extractor on ``sys.path``). Shipped in both modes so an
-# operator who flips an existing deploy from ai-fallback to
-# compiled-only doesn't need to also re-stage extractor code.
-cp "${ARTIFACTS_DIR}/reference_extractor.py" "$STAGING/"
+if [[ -n "$PROPERTY_GRAPH" ]]; then
+  # Schema-derived mode: stage only the property graph + its companion
+  # table DDL. The runtime derives ontology + binding from them, so no
+  # ontology.yaml / binding.yaml / reference_extractor.py is needed.
+  cp "$PROPERTY_GRAPH" "$STAGING/property_graph.sql"
+  cp "$TABLE_DDL_SRC" "$STAGING/table_ddl.sql"
+else
+  # Explicit ontology + binding (migration-v5 / compiled-extractor path).
+  cp "${ARTIFACTS_DIR}/ontology.yaml" "$STAGING/"
+  cp "${ARTIFACTS_DIR}/binding.yaml" "$STAGING/"
+  cp "${ARTIFACTS_DIR}/table_ddl.sql" "$STAGING/"
+  # Stage the reference extractor module next to ``run_job.py`` so
+  # Python can import it via ``BQAA_REFERENCE_EXTRACTORS_MODULE=
+  # reference_extractor`` (Buildpacks sets the container working
+  # directory to the staging dir's contents, which puts the
+  # extractor on ``sys.path``). Shipped in both modes so an
+  # operator who flips an existing deploy from ai-fallback to
+  # compiled-only doesn't need to also re-stage extractor code.
+  cp "${ARTIFACTS_DIR}/reference_extractor.py" "$STAGING/"
+fi
 
 # Vendor the local SDK source.
 mkdir -p "$STAGING/sdk_src/src"
@@ -667,6 +725,11 @@ fi
 if [[ "$EXTRACTION_MODE" == "compiled-only" ]]; then
   ENV_VARS+=("BQAA_REFERENCE_EXTRACTORS_MODULE=reference_extractor")
 fi
+# Schema-derived mode: tell the runtime to derive the spec from the staged
+# property graph instead of the explicit ontology/binding pair (#286).
+if [[ -n "$PROPERTY_GRAPH" ]]; then
+  ENV_VARS+=("BQAA_PROPERTY_GRAPH=property_graph.sql")
+fi
 # Orphan-session watchdog (issue #180). Only wired when the
 # operator explicitly opts in via ``--max-session-age-hours``;
 # default deploys ship without the watchdog so they don't pay
diff --git a/tests/test_deploy_property_graph_mode.py b/tests/test_deploy_property_graph_mode.py
@@ -0,0 +1,161 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Deploy-script property-graph mode: boundary validation + staging contract.
+
+Static + shell tests for the Cloud Run deploy/build scripts (issue #286, PR 3).
+The validation runs before any gcloud call, so the rejection paths can be
+exercised by invoking the script directly; the staging contract is pinned by
+asserting on the script text (no live gcloud / Docker needed).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+import shutil
+import subprocess
+
+import pytest
+
+_DEPLOY_DIR = (
+    Path(__file__).resolve().parents[1]
+    / "examples"
+    / "migration_v5"
+    / "periodic_materialization"
+)
+_DEPLOY = _DEPLOY_DIR / "deploy_cloud_run_job.sh"
+_BUILD = _DEPLOY_DIR / "build_image.sh"
+
+_REQUIRED = [
+    "--project",
+    "p",
+    "--region",
+    "us-central1",
+    "--events-dataset",
+    "e",
+    "--graph-dataset",
+    "g",
+    "--schedule",
+    "0 */6 * * *",
+]
+
+
+def _run_deploy(args):
+  return subprocess.run(
+      ["bash", str(_DEPLOY), *_REQUIRED, *args],
+      capture_output=True,
+      text=True,
+      timeout=60,
+  )
+
+
+@pytest.mark.skipif(shutil.which("bash") is None, reason="bash not available")
+def test_rejects_property_graph_with_compiled_only(tmp_path) -> None:
+  (tmp_path / "property_graph.sql").write_text("x")
+  (tmp_path / "table_ddl.sql").write_text("x")
+  result = _run_deploy(
+      [
+          "--property-graph",
+          str(tmp_path / "property_graph.sql"),
+          "--extraction-mode",
+          "compiled-only",
+      ]
+  )
+  assert result.returncode != 0
+  assert "compiled-only" in result.stderr
+
+
+@pytest.mark.skipif(shutil.which("bash") is None, reason="bash not available")
+def test_rejects_property_graph_without_sibling_table_ddl(tmp_path) -> None:
+  (tmp_path / "property_graph.sql").write_text("x")  # no table_ddl.sql sibling
+  result = _run_deploy(
+      ["--property-graph", str(tmp_path / "property_graph.sql")]
+  )
+  assert result.returncode != 0
+  assert "table_ddl.sql" in result.stderr
+
+
+@pytest.mark.skipif(shutil.which("bash") is None, reason="bash not available")
+def test_rejects_missing_property_graph_file(tmp_path) -> None:
+  result = _run_deploy(
+      ["--property-graph", str(tmp_path / "does_not_exist.sql")]
+  )
+  assert result.returncode != 0
+  assert "not found" in result.stderr
+
+
+@pytest.mark.skipif(shutil.which("bash") is None, reason="bash not available")
+def test_rejects_hardcoded_property_graph_without_placeholders(
+    tmp_path,
+) -> None:
+  # The original #286 failure mode: a hardcoded graph DDL (no
+  # ${PROJECT_ID}/${DATASET}) would derive against the wrong dataset. Both
+  # files exist and mode is ai-fallback, so this must be caught by the
+  # placeholder check, not the earlier existence checks.
+  (tmp_path / "property_graph.sql").write_text(
+      "CREATE PROPERTY GRAPH `proj.ds.g` NODE TABLES ()"
+  )
+  (tmp_path / "table_ddl.sql").write_text(
+      "CREATE TABLE `${PROJECT_ID}.${DATASET}.t` (id STRING)"
+  )
+  result = _run_deploy(
+      ["--property-graph", str(tmp_path / "property_graph.sql")]
+  )
+  assert result.returncode != 0
+  assert "${PROJECT_ID}" in result.stderr or "placeholder" in result.stderr
+
+
+@pytest.mark.skipif(shutil.which("bash") is None, reason="bash not available")
+def test_rejects_hardcoded_table_ddl_without_placeholders(tmp_path) -> None:
+  # The graph is placeholdered but the companion table DDL is not -> still a
+  # wrong-dataset risk at bootstrap time -> reject.
+  (tmp_path / "property_graph.sql").write_text(
+      "CREATE PROPERTY GRAPH `${PROJECT_ID}.${DATASET}.g` NODE TABLES ()"
+  )
+  (tmp_path / "table_ddl.sql").write_text(
+      "CREATE TABLE `proj.ds.t` (id STRING)"
+  )
+  result = _run_deploy(
+      ["--property-graph", str(tmp_path / "property_graph.sql")]
+  )
+  assert result.returncode != 0
+  assert "${PROJECT_ID}" in result.stderr or "placeholder" in result.stderr
+
+
+# --------------------------------------------------------------------------- #
+# Staging / env-var contract (static text assertions)
+# --------------------------------------------------------------------------- #
+
+
+def test_deploy_script_property_graph_staging_contract() -> None:
+  text = _DEPLOY.read_text()
+  # mode arg + boundary env var
+  assert "--property-graph)" in text
+  assert "BQAA_PROPERTY_GRAPH=property_graph.sql" in text
+  # property-graph mode stages the graph + its table DDL, not ontology/binding
+  assert 'cp "$PROPERTY_GRAPH" "$STAGING/property_graph.sql"' in text
+  assert 'cp "$TABLE_DDL_SRC" "$STAGING/table_ddl.sql"' in text
+  # placeholder contract is enforced for both artifacts
+  assert "grep -qF '${PROJECT_ID}'" in text
+  assert "grep -qF '${DATASET}'" in text
+
+
+def test_build_image_property_graph_staging_contract() -> None:
+  text = _BUILD.read_text()
+  assert "--property-graph)" in text
+  assert 'cp "$PROPERTY_GRAPH" "$STAGING/property_graph.sql"' in text
+  assert 'cp "$TABLE_DDL_SRC" "$STAGING/table_ddl.sql"' in text
+  # Terraform-built images can't bake hardcoded artifacts either.
+  assert "grep -qF '${PROJECT_ID}'" in text
+  assert "grep -qF '${DATASET}'" in text
