|
| 1 | +# Cloud Run Operations (living reference) |
| 2 | + |
| 3 | +This is the single living reference for how API v1's Cloud Run deployments work. It is |
| 4 | +edited in place whenever behavior changes. Per-stage records (bootstrap commands, |
| 5 | +exit-gate evidence, one-off measurements) are append-only documents under `history/` and |
| 6 | +are never updated to match later reality. |
| 7 | + |
| 8 | +Configuration **values** (CPU, memory, scaling, secrets, service names) live in |
| 9 | +`.github/scripts/cloud_run_env.sh` and the deploy scripts — they are the source of truth |
| 10 | +and are intentionally not duplicated here. This document explains the **semantics**. |
| 11 | + |
| 12 | +## Topology |
| 13 | + |
| 14 | +- The public host `api.policyengine.org` is served by **App Engine** (bound via |
| 15 | + `gcp/dispatch.yaml`). No public traffic reaches Cloud Run until the load-balancer |
| 16 | + stages of the host cutover plan (`api-v1-pr4-cloud-run-host-cutover-execution-plan.md` |
| 17 | + in the planning folder). |
| 18 | +- Two Cloud Run services in `policyengine-api` / `us-central1`: |
| 19 | + - **`policyengine-api`** — the production candidate. Every push to master deploys a |
| 20 | + tagged `--no-traffic` revision (`stage3-prod-*`), which is promoted to the service |
| 21 | + URL after integration tests. Only CI and its own health checks call this URL today. |
| 22 | + - **`policyengine-api-staging`** — the staging track's service (split from the |
| 23 | + production service in migration PR 4, Stage 1). Staging jobs pin |
| 24 | + `CLOUD_RUN_SERVICE: policyengine-api-staging`; unit tests enforce that every |
| 25 | + service-targeting Cloud Run job carries an explicit pin and that no staging job can |
| 26 | + touch production traffic. |
| 27 | +- The Docker image name is decoupled from the service name (`CLOUD_RUN_IMAGE_NAME`): |
| 28 | + production reuses the image built by the staging track, so both tracks push and pull |
| 29 | + `policyengine-api:<sha>` regardless of service. |
| 30 | +- Every API response carries `X-PolicyEngine-Backend` (`app_engine` or `cloud_run`) for |
| 31 | + in-band backend verification during the host-cutover traffic ramps. |
| 32 | + |
| 33 | +## Startup behavior |
| 34 | + |
| 35 | +The app import is slow (~230s cold: `policyengine_api/country.py` instantiates five |
| 36 | +country packages, ~210s of it) and Cloud Run hard-caps total startup-probe time at 240s — |
| 37 | +the default TCP probe already sits at that cap, and **no probe configuration can extend |
| 38 | +it**. Two mitigations are therefore built in: |
| 39 | + |
| 40 | +- `gcp/cloud_run/start.sh` runs **gunicorn with `uvicorn.workers.UvicornWorker` and no |
| 41 | + `--preload`**: the gunicorn master binds the listen socket in milliseconds (satisfying |
| 42 | + the startup probe immediately), and the import happens in the worker after fork. |
| 43 | + `--timeout 0` is required — a worker mid-import does not heartbeat, and gunicorn's |
| 44 | + default 30s watchdog would kill it. `--keep-alive 5` and `--forwarded-allow-ips '*'` |
| 45 | + preserve the previous uvicorn keep-alive and proxy-header behavior. |
| 46 | +- Candidate deploys set `--cpu-boost` to shorten the import. |
| 47 | + |
| 48 | +Consequences to keep in mind: |
| 49 | + |
| 50 | +- An instance is "started" (port bound) before the app can answer. Requests arriving in |
| 51 | + that window queue until the worker is ready or Cloud Run's request timeout fires. This |
| 52 | + is acceptable while the services carry no public traffic; the min-instances decision in |
| 53 | + the cutover plan's Stage 3 keeps cold imports off the request path before traffic ramps. |
| 54 | +- **Cold-start and readiness measurements must be taken from `/readiness-check` turning |
| 55 | + healthy**, not from instance start or port bind — the bind time is no longer meaningful. |
| 56 | +- Readiness in CI is governed by `.github/scripts/health_check.sh` polling |
| 57 | + `/readiness-check` (defaults: 900s budget, 5s interval, 15s per-request `--max-time`; |
| 58 | + all env-overridable). |
| 59 | + |
| 60 | +## IAM and bootstrap constraints |
| 61 | + |
| 62 | +- The GitHub deploy service account holds `roles/run.developer`: it can deploy to |
| 63 | + existing services but cannot create the `allUsers` → `roles/run.invoker` binding a new |
| 64 | + service needs, and `deploy_cloud_run_candidate.sh` always deploys `--no-traffic`, which |
| 65 | + gcloud rejects on service creation. **New Cloud Run services are therefore bootstrapped |
| 66 | + manually by a project owner** (placeholder image + IAM binding + runtime service |
| 67 | + account); the first CI deploy replaces the revision entirely. See |
| 68 | + `history/pr4-stage1-staging-service-runbook.md` for the pattern. |
| 69 | +- Both services currently run as the dedicated runtime service account and share the |
| 70 | + prod-named Secret Manager secrets and the production Cloud SQL instance. Known |
| 71 | + follow-up: per-service secrets became possible once the services split; migrate in a |
| 72 | + later stage. |
| 73 | + |
| 74 | +## Verification commands |
| 75 | + |
| 76 | +```bash |
| 77 | +# Service state, runtime SA, traffic tags |
| 78 | +gcloud run services describe policyengine-api \ |
| 79 | + --project policyengine-api --region us-central1 \ |
| 80 | + --format 'value(status.url, spec.template.spec.serviceAccountName, status.traffic)' |
| 81 | + |
| 82 | +# IAM (expect allUsers -> roles/run.invoker) |
| 83 | +gcloud run services get-iam-policy policyengine-api-staging \ |
| 84 | + --project policyengine-api --region us-central1 |
| 85 | + |
| 86 | +# Readiness and backend identification |
| 87 | +curl -s -o /dev/null -w '%{http_code}\n' "$SERVICE_URL/readiness-check" |
| 88 | +curl -sI "$SERVICE_URL/readiness-check" | grep -i x-policyengine-backend |
| 89 | +``` |
| 90 | + |
| 91 | +## History index |
| 92 | + |
| 93 | +- `history/migration-pr1-baseline-runbook.md` — PR 1 baseline capture |
| 94 | +- `history/migration-pr2-fastapi-shell-runbook.md` — PR 2 FastAPI/ASGI shell |
| 95 | +- `history/migration-pr3-cloud-run-candidate-runbook.md` — PR 3 candidate/promote |
| 96 | + pipeline (documents the pre-Stage-1 single-service behavior) |
| 97 | +- `history/pr4-stage1-staging-service-runbook.md` — PR 4 Stage 1 staging service split, |
| 98 | + startup de-flake, and exit gates |
0 commit comments