Custom Data Commons — single-repo deployment

Everything needed to deploy a Custom Data Commons instance (React UI + agent sidecar + per-instance branding + Terraform infrastructure) lives in this one repo. Clone it, point it at a fresh GCP project, run DEPLOY.md, and a complete instance comes up.

Deploying for the first time to a fresh GCP project? Read DEPLOY.md. Every command is copy-pasteable; every common failure has a documented fix.

Want to run the whole stack on your laptop first? Read LOCAL_DEPLOY.md. Three terminals, ~10 minutes, no GCP involvement.

Architecture rationale lives in ./architecture/ — target-architecture.md (why), migration-plan.md (staged migration), implementation-reference.md (full spec).

Multi-instance model

One repo clone deploys one instance. Each state (India, Karnataka, California, …) is a fully independent prototype deployment in its own GCP project, with its own GCS bucket, its own Cloud SQL, its own secrets. There is no shared control plane and no shared bucket across instances.

Adding a second state = clone this repo (or branch it), edit config/ for the new state, run DEPLOY.md against a different GCP project. No cross-instance plumbing.

Layout

custom-data-commons/
├── DEPLOY.md                       # ← cloud deploy, 8 phases (read first)
├── LOCAL_DEPLOY.md                 # ← full-stack laptop replication
├── README.md                       # ← this file
│
├── ui/                             # React 19 + Vite + Tailwind 4 UI
│   ├── src/
│   │   ├── components/             # DataAgent, MetricsPage, Sidebar, SessionDrawer,
│   │   │                           # DataDownloadTool, StatVarExplorer, …
│   │   ├── hooks/                  # useSseChat, ChatSessionContext, useBranding,
│   │   │                           # useHashRoute
│   │   ├── api/dcObservations.ts
│   │   └── types/
│   ├── public/                     # logo + static PNGs
│   ├── package.json
│   ├── vite.config.ts              # /api, /mcp, /tools, /agent proxies (dev only)
│   └── tsconfig.json
│
├── agent/                          # Vendored Flask agent (mcp_proxy_only.py + 8 patches)
│   ├── mcp_proxy_only.py
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── build.sh
│   └── PATCHES.md                  # patch-by-patch rationale
│
├── image/                          # Services overlay on datacommons-services:stable
│   ├── Dockerfile                  # FROM stable + ui/dist + nginx.conf
│   ├── nginx.conf                  # /agent/* → :5001; /assets/* → React build
│   └── build.sh                    # runs `npm run build` in ../ui, then docker buildx
│
├── config/                         # THIS instance's bucket contents — mirrors gs://<project>-config/
│   ├── branding.json               # logo, colors, suggestion chips, metrics.tabs[]
│   ├── agent-config.json           # Gemini models, thinking, prompts, filestores, template_vars
│   ├── agent-config.schema.json    # JSON Schema for the above
│   ├── agent-config.example.json   # canonical example with every field
│   ├── prompts/{mcp,kb,synthesis}.md
│   ├── skills/                     # reusable prompt fragments
│   ├── assets/                     # logo.png, overrides.css, …
│   └── README.md                   # schema doc + India DCID conventions
│
├── sample-data/                    # OECD CSVs + config.json for the upstream ingest job
│
├── deploy/terraform-custom-datacommons/   # IaC
│   ├── modules/                    # main.tf, variables.tf, service_account.tf,
│   │                               # backend.tf (GCS state), data_ingest_job.tf
│   └── new-instance.tfvars.sample  # template for fresh-project deploy
│
├── cloudbuild/                     # CI pipelines
│   ├── pr-validate.yaml            # on PR: terraform fmt + tfsec + schema + dual image build
│   ├── deploy-stamp.yaml           # on merge: build + push + terraform apply + smoke
│   └── promote-image.yaml          # weekly: mirror upstream :stable → AR
│
├── docs/                           # supporting runbooks
│   ├── smoke.sh                    # 6-step post-deploy smoke
│   └── baseline-india.md           # Stage 0 inventory (prompt + asset SHAs)
│
└── architecture/                   # design docs (the "why")
    ├── target-architecture.md      # executive doc, client-facing
    ├── migration-plan.md           # staged 7-stage VM → Cloud Run plan
    ├── implementation-reference.md # full engineering spec
    ├── ui-tracker.csv              # progress tracker
    ├── diagram.py                  # source for the architecture diagram
    └── target-architecture-diagram.{png,svg,drawio}

Quick start

See DEPLOY.md for the full guide. At a glance:

# 0. Inputs
export PROJECT_ID="<your-new-gcp-project>"
export INSTANCE="<short-dns-safe-name>"      # e.g. india
export REGION="us-central1"

# 1. Project bootstrap (APIs, AR, tfstate bucket, docker auth)
# 2. Secrets (4 secret manager entries; user provides 3 API keys, generates DB_PASS)
# 3. Build the React UI + agent + services images, push to AR
# 4. Create the per-instance config bucket; gsutil rsync config/ to bucket root
# 5. Provision Cloud SQL + data ingest via Terraform; stage CSVs; run ingest job
# 6. Provision Cloud Run via Terraform; capture the service URL
# 7. Smoke test (6 endpoints + chat-stream)
# 8. Open URL in browser

Each phase has a verification block in DEPLOY.md; failures map to an F-N row in the troubleshooting table.

Config model

Everything per-instance is data, not code, and lives in one folder: config/. Customising this instance means editing files under config/ and re-running the deploy. There is no separate _framework/ directory and no per-state prefix in the bucket — config/ IS the bucket layout.

To customise:

1. Edit config/branding.json (logo, colors, suggestion chips, footer text, metrics.tabs[]).
2. Edit config/agent-config.json (Gemini model choice, thinking levels, template_vars for instance name/region/state-term, gemini.filestores[] corpus ID).
3. Replace config/assets/logo.png and any state-specific images.
4. Optionally adapt config/prompts/*.md for state-specific dataset references.
5. Re-run DEPLOY.md Phase 4 (gsutil rsync) to push the changes to the bucket — no image rebuild.

See config/README.md for the schema-level detail.

Fast deploys with deploy.sh

deploy.sh is a one-command orchestrator over the DEPLOY.md phases, with surgical flags for the inner loop:

Flag	What it does
(default)	Full greenfield deploy (APIs, secrets, images, config bucket, Terraform, ingest).
--code-only	Rebuild both containers + deploy; skip data/SQL.
--frontend-only / --agent-only	Rebuild just one container.
--infra-only	terraform apply only, using the active image tags.
--config-only	Sync config/ to the bucket and refresh the agent's branding cache — no image build, no Terraform.
--config-only --restart	Same, plus a Cloud Run restart so the agent reloads startup-only config (agent-config.json, prompts, skills).

Branding is fetched live by the agent (short TTL + /agent/brand?refresh=1), so ./deploy.sh --config-only makes a branding.json edit visible on refresh without a rebuild. See BRANDING_CONFIG_CHANGES.md for the full branding/agent-config update workflow and GCP verification steps.

Two agent config modes

• AGENT_CONFIG_MODE=dev — editable "Agent config" panel in the UI; session-scoped ?config= overlay. For prototyping and tuning. (Handover deliverable — see architecture/target-architecture.md §6.3.)
• AGENT_CONFIG_MODE=prod — panel hidden; config is agent-config.json in the bucket, version-controlled in this repo. For shipped instances.

Same JSON Schema (config/agent-config.schema.json) validates both modes.

Agent patches

The vendored agent/mcp_proxy_only.py has 8 patches over upstream additional_features/mcp_proxy_only.py from the custom-dc-setup branch. See agent/PATCHES.md for the full table.