Skip to content

Commit b0d6cfa

Browse files
anth-volkclaude
anth-volk
and
claude
committed
Add Stage 4 load-balancer runbook and weight-change procedure
Gcloud equivalents of every console action for provisioning the global external ALB (EXTERNAL_MANAGED): dual IPv4+IPv6 frontends (the current host serves AAAA via ghs.googlehosted.com, so IPv6 clients exist), DNS-authorized Certificate Manager cert for both hostnames, serverless NEGs for App Engine and Cloud Run in us-central1, backend services with timeout 600 and full request logging, and a URL map pinned to app_engine=100/cloud_run=0 so every future ramp step is a weight edit. Committed before execution per the plan's manual-infra rule; exit gates and full teardown included. The recurring weight-change procedure (export -> diff -> edit -> import -> verify -> commit snapshot) lands in the living operations doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent b2819d5 commit b0d6cfa

3 files changed

Lines changed: 244 additions & 0 deletions

File tree

‎changelog.d/pr4-stage4-lb-runbook.added.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add the Stage 4 load-balancer provisioning runbook and the URL-map weight-change procedure for the Cloud Run host cutover. Documentation only; no infrastructure is created by this change.

‎docs/migration/cloud-run-operations.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,28 @@ curl -s -o /dev/null -w '%{http_code}\n' "$SERVICE_URL/readiness-check"
137137
curl -sI "$SERVICE_URL/readiness-check" | grep -i x-policyengine-backend
138138
```
139139

140+
## URL map weight changes (the ramp lever, Stages 5 and 9–11)
141+
142+
Once the load balancer exists (Stage 4), every traffic shift between App Engine and
143+
Cloud Run is a weight edit on the `lb-api` URL map — and **only** via this procedure.
144+
Console edits leave no audit trail, and `url-maps import` **replaces the entire map**,
145+
so an un-diffed import can silently drop routing config:
146+
147+
```bash
148+
gcloud compute url-maps export lb-api --project policyengine-api \
149+
--destination docs/migration/urlmap/$(date -u +%Y%m%dT%H%M%SZ)-lb-api.yaml
150+
# 1. diff against the last committed snapshot — investigate ANY unexpected delta
151+
# 2. edit ONLY the two weight values (must sum to 100)
152+
# 3. import, then re-export and verify the readback matches the edit
153+
gcloud compute url-maps import lb-api --project policyengine-api --source <edited>.yaml --quiet
154+
# 4. verify in-band before trusting it: ~20 header-sampled curls against the LB
155+
# counting X-PolicyEngine-Backend values
156+
# 5. commit the new snapshot to docs/migration/urlmap/
157+
```
158+
159+
Rollback during a ramp = the same procedure with `app_engine=100, cloud_run=0`
160+
(measured propagation is minutes; rehearsed in Stage 9).
161+
140162
## History index
141163

142164
- `history/migration-pr1-baseline-runbook.md` — PR 1 baseline capture

‎docs/migration/history/pr4-stage4-load-balancer-runbook.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# PR 4 Stage 4 Runbook: Provision the Load Balancer
2+
3+
> Per-stage record: gcloud equivalents of every console action, committed **before**
4+
> manual execution (the plan's infra rule). Current operational behavior:
5+
> [`../cloud-run-operations.md`](../cloud-run-operations.md). Source of truth for stage
6+
> gates: the Stage 4 section of `api-v1-pr4-cloud-run-host-cutover-execution-plan.md`
7+
> in the planning folder.
8+
9+
Builds the global external Application Load Balancer (`EXTERNAL_MANAGED`) that will
10+
eventually front `api.policyengine.org`, with two serverless NEG backends (App Engine,
11+
Cloud Run) and a weighted URL map pinned to **app_engine=100 / cloud_run=0**. This stage
12+
creates infrastructure only — **no DNS change to `api.policyengine.org`** (the only DNS
13+
edits are the two certificate-authorization CNAMEs; the `api-lb` validation hostname is
14+
Stage 5, the real cutover is Stage 8).
15+
16+
Facts baked in from pre-checks (2026-07-08):
17+
18+
- `api.policyengine.org` currently CNAMEs to `ghs.googlehosted.com`, which serves **both
19+
A and AAAA** — IPv6 clients exist today, so the LB gets dual (IPv4 + IPv6) frontends.
20+
- The App Engine app's `locationId` is `us-central` → serverless NEGs go in compute
21+
region **`us-central1`** (same region as the Cloud Run service).
22+
- Executor needs `roles/compute.loadBalancerAdmin` + `roles/certificatemanager.editor`
23+
(or owner) on `policyengine-api`; the CI deploy service account has neither and never
24+
will — this stage is manual by design.
25+
26+
## 0. Shell variables (paste first)
27+
28+
```bash
29+
export PROJECT=policyengine-api REGION=us-central1
30+
export IP4=policyengine-api-lb-ip IP6=policyengine-api-lb-ipv6
31+
export NEG_GAE=neg-app-engine NEG_CR=neg-cloud-run
32+
export BS_GAE=bs-app-engine BS_CR=bs-cloud-run
33+
export MAP=lb-api REDIRECT_MAP=lb-api-redirect
34+
export PROXY_HTTPS=proxy-api-https PROXY_HTTP=proxy-api-http
35+
export CERT=cert-api-lb CERT_MAP=map-api
36+
```
37+
38+
## 1. Static IPs
39+
40+
```bash
41+
gcloud compute addresses create "$IP4" --project "$PROJECT" --global --ip-version=IPV4
42+
gcloud compute addresses create "$IP6" --project "$PROJECT" --global --ip-version=IPV6
43+
gcloud compute addresses list --project "$PROJECT" --global \
44+
--format 'table(name, address)' # record BOTH in this file when executed
45+
```
46+
47+
Executed values: `IP4 = ______` / `IP6 = ______` (fill in at execution).
48+
49+
## 2. Certificate via DNS authorization (both hostnames)
50+
51+
```bash
52+
for D in api api-lb; do
53+
gcloud certificate-manager dns-authorizations create "auth-${D}" \
54+
--project "$PROJECT" --domain "${D}.policyengine.org"
55+
done
56+
gcloud certificate-manager dns-authorizations list --project "$PROJECT" \
57+
--format 'table(name, dnsResourceRecord.name, dnsResourceRecord.data)'
58+
```
59+
60+
Add the two CNAME records printed above **at Squarespace** (they look like
61+
`_acme-challenge.api.policyengine.org → xxxx.authorize.certificatemanager.goog`).
62+
These are additive records — they do not affect serving DNS. Then:
63+
64+
```bash
65+
gcloud certificate-manager certificates create "$CERT" --project "$PROJECT" \
66+
--domains="api.policyengine.org,api-lb.policyengine.org" \
67+
--dns-authorizations="auth-api,auth-api-lb"
68+
gcloud certificate-manager maps create "$CERT_MAP" --project "$PROJECT"
69+
for D in api api-lb; do
70+
gcloud certificate-manager maps entries create "entry-${D}" --project "$PROJECT" \
71+
--map "$CERT_MAP" --hostname "${D}.policyengine.org" --certificates "$CERT"
72+
done
73+
# GATE (may take minutes after the CNAMEs propagate):
74+
gcloud certificate-manager certificates describe "$CERT" --project "$PROJECT" \
75+
--format 'value(managed.state, managed.authorizationAttemptInfo)'
76+
# expect state ACTIVE and both domains AUTHORIZED before proceeding to step 5.
77+
```
78+
79+
## 3. Serverless NEGs
80+
81+
```bash
82+
gcloud compute network-endpoint-groups create "$NEG_GAE" --project "$PROJECT" \
83+
--region "$REGION" --network-endpoint-type=serverless --app-engine-service=default
84+
gcloud compute network-endpoint-groups create "$NEG_CR" --project "$PROJECT" \
85+
--region "$REGION" --network-endpoint-type=serverless --cloud-run-service=policyengine-api
86+
```
87+
88+
Pinning `--app-engine-service=default` is deliberate: the NEG routes straight to the
89+
service, **bypassing `gcp/dispatch.yaml`** — the dispatch file stays as the rollback
90+
path for the direct App Engine domain mapping, not part of the LB path.
91+
92+
## 4. Backend services (timeout 600, full request logging)
93+
94+
```bash
95+
for PAIR in "$BS_GAE:$NEG_GAE" "$BS_CR:$NEG_CR"; do
96+
BS="${PAIR%%:*}"; NEG="${PAIR##*:}"
97+
gcloud compute backend-services create "$BS" --project "$PROJECT" --global \
98+
--load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTPS \
99+
--timeout=600 --enable-logging --logging-sample-rate=1.0
100+
gcloud compute backend-services add-backend "$BS" --project "$PROJECT" --global \
101+
--network-endpoint-group="$NEG" --network-endpoint-group-region="$REGION"
102+
done
103+
```
104+
105+
`--timeout=600`: the default 30s would kill long `/calculate` requests; Cloud Run's own
106+
request cap is 300s and App Engine's is longer, so 600 makes the LB never the binding
107+
constraint. **Validated empirically in Stage 5's long-request proof** — if the timeout
108+
turns out not to be honored for serverless NEGs, that gate catches it before any DNS
109+
change. Reminder: serverless NEGs have **no health checks** — alerting (Stage 6) is the
110+
failover mechanism.
111+
112+
## 5. URL map at 100/0, proxies, forwarding rules
113+
114+
```bash
115+
gcloud compute url-maps create "$MAP" --project "$PROJECT" \
116+
--default-service "$BS_GAE"
117+
gcloud compute url-maps export "$MAP" --project "$PROJECT" \
118+
--destination /tmp/lb-api.yaml
119+
```
120+
121+
Edit `/tmp/lb-api.yaml`: replace the top-level `defaultService: ...` line with the
122+
weighted action (weights are the ramp lever for Stages 9–11):
123+
124+
```yaml
125+
defaultRouteAction:
126+
weightedBackendServices:
127+
- backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine
128+
weight: 100
129+
- backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run
130+
weight: 0
131+
```
132+
133+
```bash
134+
gcloud compute url-maps import "$MAP" --project "$PROJECT" --source /tmp/lb-api.yaml --quiet
135+
```
136+
137+
Then the HTTPS side (certificate map attaches to the proxy — no `--ssl-certificates`):
138+
139+
```bash
140+
gcloud compute target-https-proxies create "$PROXY_HTTPS" --project "$PROJECT" \
141+
--url-map "$MAP" --certificate-map "$CERT_MAP"
142+
for PAIR in "fr-api-https:$IP4" "fr-api-https-v6:$IP6"; do
143+
FR="${PAIR%%:*}"; ADDR="${PAIR##*:}"
144+
gcloud compute forwarding-rules create "$FR" --project "$PROJECT" --global \
145+
--load-balancing-scheme=EXTERNAL_MANAGED --address="$ADDR" \
146+
--target-https-proxy="$PROXY_HTTPS" --ports=443
147+
done
148+
```
149+
150+
HTTP→HTTPS 301 (separate redirect map; `url-maps create` has no redirect flags, so
151+
import it):
152+
153+
```bash
154+
cat > /tmp/lb-api-redirect.yaml <<'YAML'
155+
name: lb-api-redirect
156+
defaultUrlRedirect:
157+
httpsRedirect: true
158+
redirectResponseCode: MOVED_PERMANENTLY_DEFAULT
159+
stripQuery: false
160+
YAML
161+
gcloud compute url-maps import "$REDIRECT_MAP" --project "$PROJECT" \
162+
--source /tmp/lb-api-redirect.yaml --quiet
163+
gcloud compute target-http-proxies create "$PROXY_HTTP" --project "$PROJECT" \
164+
--url-map "$REDIRECT_MAP"
165+
for PAIR in "fr-api-http:$IP4" "fr-api-http-v6:$IP6"; do
166+
FR="${PAIR%%:*}"; ADDR="${PAIR##*:}"
167+
gcloud compute forwarding-rules create "$FR" --project "$PROJECT" --global \
168+
--load-balancing-scheme=EXTERNAL_MANAGED --address="$ADDR" \
169+
--target-http-proxy="$PROXY_HTTP" --ports=80
170+
done
171+
```
172+
173+
## 6. Snapshot the URL map (audit trail for every future weight change)
174+
175+
```bash
176+
mkdir -p docs/migration/urlmap
177+
gcloud compute url-maps export "$MAP" --project "$PROJECT" \
178+
--destination "docs/migration/urlmap/$(date -u +%Y%m%dT%H%M%SZ)-lb-api.yaml"
179+
git add docs/migration/urlmap/ && git commit -m "Snapshot LB URL map (Stage 4 initial, 100/0)"
180+
```
181+
182+
All future weight changes follow the procedure in
183+
[`../cloud-run-operations.md`](../cloud-run-operations.md) ("URL map weight changes"):
184+
export → diff → edit → import → commit. Console clicks leave no audit trail, and
185+
`url-maps import` **replaces the whole map** — always diff first.
186+
187+
## Exit gates (from the plan; evaluate before Stage 5)
188+
189+
```bash
190+
LB_IP=$(gcloud compute addresses describe "$IP4" --project "$PROJECT" --global --format 'value(address)')
191+
curl -sv --resolve "api.policyengine.org:443:${LB_IP}" \
192+
https://api.policyengine.org/liveness-check -o /dev/null 2>&1 \
193+
| grep -E "HTTP/|x-policyengine-backend"
194+
# expect: 200 + X-PolicyEngine-Backend: app_engine (traffic reaches GAE through the LB)
195+
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://${LB_IP}/liveness-check" \
196+
-H 'Host: api.policyengine.org'
197+
# expect: 301 → https://api.policyengine.org/liveness-check
198+
```
199+
200+
- Cert `state: ACTIVE`, both hostnames `AUTHORIZED` (step 2).
201+
- LB request logs visible in Logging with `resource.type="http_load_balancer"` and
202+
`backend_service_name` populated (allow a few minutes after first traffic).
203+
- IPv6 spot check: repeat the `--resolve` curl against the IPv6 address.
204+
205+
## Teardown (full rollback of this stage; reverse order)
206+
207+
```bash
208+
gcloud compute forwarding-rules delete fr-api-https fr-api-https-v6 fr-api-http fr-api-http-v6 --global --project "$PROJECT" --quiet
209+
gcloud compute target-https-proxies delete "$PROXY_HTTPS" --project "$PROJECT" --quiet
210+
gcloud compute target-http-proxies delete "$PROXY_HTTP" --project "$PROJECT" --quiet
211+
gcloud compute url-maps delete "$MAP" "$REDIRECT_MAP" --project "$PROJECT" --quiet
212+
gcloud compute backend-services delete "$BS_GAE" "$BS_CR" --global --project "$PROJECT" --quiet
213+
gcloud compute network-endpoint-groups delete "$NEG_GAE" "$NEG_CR" --region "$REGION" --project "$PROJECT" --quiet
214+
gcloud compute addresses delete "$IP4" "$IP6" --global --project "$PROJECT" --quiet
215+
# certificate-manager resources can stay (no cost, reusable) or be deleted last.
216+
```
217+
218+
No serving-path risk at any point in this stage: nothing resolves to the LB until the
219+
Stage 5 `api-lb` record, and `api.policyengine.org` continues to point at App Engine
220+
until Stage 8. Standing cost while provisioned: ~$0.025/h per forwarding rule
221+
(4 rules ≈ $73/month) plus egress — starts as soon as step 5 completes.

0 commit comments

Comments
 (0)