Skip to content

Commit 69d1a18

Browse files
mastermanas805claude
mastermanas805
and
claude
committed
k8s: sync app/worker/provisioner manifests with live prod state
Codify the live prod state of three deployment manifests after a near-miss 2026-05-20 where a naive kubectl apply against the repo's k8s/app.yaml would have: - stripped imagePullSecrets (ghcr-pull, ghcr-org-pull) leaving pods in ImagePullBackOff - reset image to instant-api:local (no such image in prod) - added a wait-for-platform-db initContainer that blocks forever in prod (DO Managed Postgres has no in-cluster postgres-platform Service) - dropped ~15 env vars (RAZORPAY_PLAN_ID_*_YEARLY, OBJECT_STORE_*, GITHUB_CLIENT_ID, GOOGLE_CLIENT_ID, BUILD_IMAGE_REGISTRY, DEPLOY_DOMAIN, CERT_ISSUER, E2E_TEST_TOKEN, OTEL_EXPORTER_OTLP_HEADERS) Per CLAUDE.md rule 15: this repo has no auto-apply by design. The fix codifies the live structure (imagePullSecrets, terminationGracePeriodSeconds, lifecycle preStop, the full env-var list) and replaces the hardcoded instant-*:local image with a ghcr.io/instanode-dev/<name>:placeholder fallback — a literal apply will now ErrImagePull loudly instead of silently regressing to a stale image. The auto-deploy CI in each backend repo overwrites the placeholder via kubectl set image on every push. The wait-for-* initContainers were removed from the base manifests since prod uses managed Postgres (no in-cluster Service); local dev should use an overlay, documented in the new k8s/APPLY-CHECKLIST.md. OTEL_EXPORTER_OTLP_HEADERS / CUSTOMER_DATABASE_URL / CUSTOMER_REDIS_URL / MONGO_ADMIN_URI / POSTGRES_CUSTOMERS_URL: live has these as inline value literals with embedded credentials/license keys. The codified intent (commented blocks) is to pull from instant-{secrets,infra-secrets}. The one-time reconciliation procedure (kubectl patch deploy --type=json to remove the inline entries, then apply) is documented in APPLY-CHECKLIST.md. Server-side dry-run: all three manifests now apply cleanly (image-tag drift only — exactly what CI auto-deploy is meant to override). Files: k8s/app.yaml: +imagePullSecrets, -wait-for-platform-db, +placeholder image, +15 env vars (yearly Razorpay, deploy backend, OAuth client IDs, object-store, E2E token; OTLP headers as commented intent) k8s/worker/deployment.yaml: +imagePullSecrets (ghcr-pull, ghcr-regrade), +terminationGracePeriodSeconds:30, +volumes/volumeMount for plans ConfigMap, +envFrom instant-infra-secrets, +9 env vars (worker pg pool, email provider, plans path, internal api URL, Razorpay, object store; CUSTOMER_* + OTLP headers as commented intent) k8s/provisioner/deployment.yaml: +imagePullSecrets, -wait-for- provisioner-db, +placeholder image, +6 env vars (public-host DNS, redis routes, Mongo/Queue backend = k8s; OTLP headers as commented intent). POSTGRES_CUSTOMERS_URL now sourced from secret. k8s/APPLY-CHECKLIST.md: new runbook documenting the hard rules, standard pre-apply procedure, and the 2026-05-20 near-miss in detail. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 684d9c4 commit 69d1a18

4 files changed

Lines changed: 577 additions & 31 deletions

File tree

‎k8s/APPLY-CHECKLIST.md‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
# Apply Checklist — Deployment Manifests
2+
3+
> Codified 2026-05-20 after a near-miss where applying the repo's
4+
> `app.yaml` would have stripped `imagePullSecrets`, reset the image to
5+
> `instant-api:local`, and added a `wait-for-platform-db` init container
6+
> that blocks forever in prod. See "What went wrong" at the bottom.
7+
8+
This checklist applies to:
9+
10+
- `infra/k8s/app.yaml` — `Deployment/instant-api` in namespace `instant`
11+
- `infra/k8s/worker/deployment.yaml` — `Deployment/instant-worker` in `instant-infra`
12+
- `infra/k8s/provisioner/deployment.yaml` — `Deployment/instant-provisioner` in `instant-infra`
13+
14+
Per CLAUDE.md rule 15: **this repo has no auto-apply by design.** Manifest
15+
apply is a deliberate, human-driven step.
16+
17+
---
18+
19+
## Hard rules
20+
21+
1. **DO NOT `kubectl apply -f k8s/app.yaml` against prod without first
22+
running `kubectl apply --dry-run=server` and reading the diff line by
23+
line.** The dry-run reveals exactly what the apply will change. If the
24+
diff includes anything other than the image tag, STOP and investigate.
25+
26+
2. **`imagePullSecrets` MUST be present** on every deployment manifest:
27+
28+
| Deployment | Required secrets |
29+
|---|---|
30+
| `instant-api` | `ghcr-pull`, `ghcr-org-pull` |
31+
| `instant-worker` | `ghcr-pull`, `ghcr-regrade` |
32+
| `instant-provisioner` | `ghcr-pull`, `ghcr-regrade` |
33+
34+
Without these, new pods land in `ImagePullBackOff`. The auto-deploy CI
35+
(which sets the image via `kubectl set image`) cannot repair stripped
36+
`imagePullSecrets` — it only touches the image field.
37+
38+
3. **The image tag in this file is a placeholder.** The actual prod image
39+
is set by the per-service auto-deploy CI in the api/worker/provisioner
40+
repos via:
41+
42+
```
43+
kubectl set image deploy/<name> <container>=ghcr.io/instanode-dev/<image>:master-<sha>
44+
```
45+
46+
The placeholder `:placeholder` is intentional — if a literal apply
47+
ever reaches prod, the pull will fail loudly (`ErrImagePull`) instead
48+
of silently regressing to a stale image. Loud failure > silent regression.
49+
50+
4. **Init containers that reference in-cluster services that don't
51+
exist in prod are removed from the base manifest.** The legacy
52+
`wait-for-platform-db` (and provisioner's `wait-for-provisioner-db`)
53+
init containers expected an in-cluster `postgres-platform` /
54+
`postgres-provisioner` Service. Prod uses DigitalOcean Managed
55+
Postgres — those Services do not exist. Init containers would block
56+
pod startup indefinitely.
57+
58+
For local dev (Rancher Desktop / k3s with an in-cluster postgres pod),
59+
layer a kustomize overlay or just patch the init container in by hand:
60+
61+
```
62+
kubectl patch deploy/instant-api -n instant --type=json -p='[
63+
{"op":"add","path":"/spec/template/spec/initContainers","value":[
64+
{"name":"wait-for-platform-db","image":"postgres:16-alpine",
65+
"command":["sh","-c","until pg_isready -h postgres-platform -U instant -d instant_platform; do sleep 2; done"]}
66+
]}
67+
]'
68+
```
69+
70+
5. **`terminationGracePeriodSeconds`, `lifecycle.preStop`, and graceful-
71+
shutdown probes must stay codified** (MR-P0-7, 2026-05-20). The api
72+
needs 35s (5s preStop + 3s readiness drain + 25s Fiber drain + 2s
73+
buffer) — anything less and k8s SIGKILLs mid-drain. The worker drains
74+
River jobs in `Workers.Stop` (MR-P0-2/P1-3) and needs the default 30s.
75+
76+
6. **`E2E_TEST_TOKEN` is declared ONCE on `instant-api`.** Earlier live
77+
manifests had two `env[E2E_TEST_TOKEN]` entries — kubectl raised a
78+
`hides previous definition` warning. The repo manifest declares it
79+
exactly once with `optional: true`.
80+
81+
---
82+
83+
## Standard pre-apply procedure
84+
85+
```bash
86+
# 1. Confirm context
87+
kubectl config current-context
88+
# Expected: do-nyc3-instant-prod (for prod) or rancher-desktop (for local)
89+
90+
# 2. Dry-run server-side (validates against the real API server, surfaces
91+
# schema errors AND shows what would change without changing anything)
92+
kubectl apply --dry-run=server -f k8s/app.yaml
93+
94+
# 3. Diff against live (this is the SOURCE OF TRUTH for what apply will do)
95+
diff <(kubectl apply --dry-run=server -f k8s/app.yaml -o yaml 2>/dev/null) \
96+
<(kubectl get deploy/instant-api -n instant -o yaml)
97+
98+
# 4. Read every line of the diff. Acceptable drift:
99+
# - image tag (will be overwritten by the next auto-deploy CI run)
100+
# - status block (read-only, computed by the controller)
101+
# - metadata.resourceVersion, generation (managed by the API server)
102+
# Anything else → STOP, investigate.
103+
104+
# 5. If the diff is clean, apply
105+
kubectl apply -f k8s/app.yaml
106+
107+
# 6. After apply, immediately re-trigger the auto-deploy CI in the
108+
# api/worker/provisioner repo (or `kubectl set image` manually) to
109+
# restore the real image tag.
110+
```
111+
112+
---
113+
114+
## What goes in the manifest vs the cluster
115+
116+
The manifest is the **structural** source of truth: deployment shape,
117+
container names, env var declarations (with `valueFrom` references),
118+
probes, resource limits, imagePullSecrets, volumeMounts.
119+
120+
The cluster is the **value** source of truth for secrets and the
121+
current image tag:
122+
123+
- All `secretKeyRef`-resolved env vars get their values from
124+
`instant-secrets` / `instant-infra-secrets` (live in the cluster, NOT
125+
in this repo — see `secrets.yaml` template warning in `README.md`).
126+
- The image tag is owned by the per-service auto-deploy CI on push to
127+
master in each backend repo.
128+
129+
If you add a new `secretKeyRef` env var here, you must ALSO
130+
`kubectl patch secret instant-secrets ...` (or `instant-infra-secrets`)
131+
with the real value — the manifest only declares the reference, it
132+
doesn't seed the secret.
133+
134+
---
135+
136+
## What went wrong (2026-05-20 near-miss)
137+
138+
The repo's `app.yaml` had drifted away from live prod over several
139+
weeks of incremental `kubectl patch` and `kubectl set image` operations
140+
that never made it back into the file:
141+
142+
| Drift | Repo before fix | Live |
143+
|---|---|---|
144+
| `imagePullSecrets` | absent | `ghcr-pull`, `ghcr-org-pull` |
145+
| Container image | `instant-api:local` | `ghcr.io/instanode-dev/instant-api:master-<sha>` |
146+
| `wait-for-platform-db` initContainer | present | absent (DO Managed Postgres) |
147+
| `OTEL_EXPORTER_OTLP_HEADERS` env | absent | present (NR license key) |
148+
| `BUILD_IMAGE_REGISTRY` / `DEPLOY_DOMAIN` / `CERT_ISSUER` env | absent | present |
149+
| `RAZORPAY_PLAN_ID_*_YEARLY` env | absent | present |
150+
| `OBJECT_STORE_*` env (7 keys) | absent | present |
151+
| `GITHUB_CLIENT_ID` / `GOOGLE_CLIENT_ID` env | absent | present |
152+
| `E2E_TEST_TOKEN` env | absent (live had 2 — duplication warning) | present (deduped here, optional: true) |
153+
154+
A naive `kubectl apply -f k8s/app.yaml` would have:
155+
156+
1. Stripped `imagePullSecrets` → new pods stuck in `ImagePullBackOff`
157+
2. Reset image to `instant-api:local` → ImagePullBackOff (no such image)
158+
3. Added the `wait-for-platform-db` init container → pod blocks forever
159+
waiting for an in-cluster service that doesn't exist
160+
4. Dropped ~15 env vars → api boots but immediately fails on missing
161+
Razorpay annual plans, object-store creds, OAuth client IDs, OTLP
162+
headers, etc.
163+
164+
That is, three independent failure modes per pod. With `replicas: 2` and
165+
rolling update, that's enough to wipe the api fleet within a minute.
166+
167+
The fix codifies the live state. The placeholder image tag (`:placeholder`)
168+
guarantees that a future naive apply fails loudly (`ErrImagePull`) instead
169+
of silently regressing.
170+
171+
---
172+
173+
## Related files
174+
175+
- `README.md` — secrets clobber warning (the same class of bug, but for
176+
the `secrets.yaml` template)
177+
- `scripts/safe-secret-apply.sh` — runtime guardrail against
178+
`CHANGE_ME`-clobbering applies of secret YAMLs
179+
- `docs/IMAGE-RETENTION-POLICY.md` — image pinning and retention policy
180+
referenced by the `instanode.dev/image-pinned` labels
181+
- `apply-all.sh` — the bootstrap script (intended for fresh clusters,
182+
NOT for in-place prod updates)

‎k8s/app.yaml‎

Lines changed: 162 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -52,20 +52,34 @@ spec:
5252
matchLabels:
5353
app: instant-api
5454
topologyKey: kubernetes.io/hostname
55-
initContainers:
56-
# Wait for platform DB to be ready before starting the API
57-
- name: wait-for-platform-db
58-
image: postgres:16-alpine
59-
command:
60-
- sh
61-
- -c
62-
- |
63-
until pg_isready -h postgres-platform -U instant -d instant_platform; do
64-
echo "Waiting for platform DB..."; sleep 2
65-
done
55+
# imagePullSecrets — required to pull the prod image from GHCR org
56+
# `instanode-dev`. Removing either of these will leave new pods stuck
57+
# in ImagePullBackOff after the next rollout. The auto-deploy CI sets
58+
# the image tag via `kubectl set image` but does NOT (and cannot)
59+
# repair imagePullSecrets if they get stripped by a naive
60+
# `kubectl apply -f app.yaml` — see APPLY-CHECKLIST.md.
61+
imagePullSecrets:
62+
- name: ghcr-pull
63+
- name: ghcr-org-pull
64+
# NOTE — wait-for-platform-db initContainer intentionally absent.
65+
# Prod runs against DigitalOcean Managed Postgres (no in-cluster
66+
# `postgres-platform` Service); the init container would block
67+
# pod startup forever. For local dev (Rancher Desktop / k3s with
68+
# an in-cluster postgres-platform Service), use the
69+
# `local-dev-overlay.yaml` overlay alongside this file. See
70+
# APPLY-CHECKLIST.md.
6671
containers:
6772
- name: api
68-
image: instant-api:local # built with: docker build -t instant-api:local .
73+
# Placeholder image — the auto-deploy CI (api repo's
74+
# .github/workflows/deploy.yml) overrides this via
75+
# `kubectl set image deploy/instant-api api=ghcr.io/instanode-dev/instant-api:master-<sha>`
76+
# on every push to master. The value below is a fallback that
77+
# documents intent + lets `kubectl apply --dry-run` succeed; the
78+
# tag `placeholder` will fail to pull if a literal apply ever
79+
# reaches the cluster, which is the LOUD failure we want over
80+
# a SILENT regression to `instant-api:local`. Per CLAUDE.md
81+
# rule 15, this repo has no auto-apply by design.
82+
image: ghcr.io/instanode-dev/instant-api:placeholder # see APPLY-CHECKLIST.md
6983
# preStop — sleep 5s before SIGTERM is delivered so the kubelet
7084
# has a tick to observe the readinessProbe failure (the api
7185
# flips inside runServerWithGracefulShutdown via
@@ -192,6 +206,112 @@ spec:
192206
name: instant-secrets
193207
key: MINIO_ROOT_PASSWORD
194208
optional: true
209+
# ── Razorpay annual plan IDs (codified 2026-05-20) ────────────────
210+
# Live since the annual-pricing toggle shipped. Required for any
211+
# *_yearly plan checkout to succeed. NOT optional in prod, but
212+
# marked so to let the api boot in a fresh cluster before secrets
213+
# are populated.
214+
- name: RAZORPAY_PLAN_ID_HOBBY_YEARLY
215+
valueFrom:
216+
secretKeyRef:
217+
name: instant-secrets
218+
key: RAZORPAY_PLAN_ID_HOBBY_YEARLY
219+
optional: true
220+
- name: RAZORPAY_PLAN_ID_PRO_YEARLY
221+
valueFrom:
222+
secretKeyRef:
223+
name: instant-secrets
224+
key: RAZORPAY_PLAN_ID_PRO_YEARLY
225+
optional: true
226+
# ── Deploy backend (the /deploy/new wedge) ────────────────────────
227+
# BUILD_IMAGE_REGISTRY is where Kaniko pushes the built customer
228+
# image; DEPLOY_DOMAIN is the apex DNS for *.deployment.<>;
229+
# CERT_ISSUER is the cert-manager ClusterIssuer that mints TLS
230+
# certs for each deployed app's ingress.
231+
- name: BUILD_IMAGE_REGISTRY
232+
value: "ghcr.io/REDACTED/instant-userapp"
233+
- name: DEPLOY_DOMAIN
234+
value: "deployment.instanode.dev"
235+
- name: CERT_ISSUER
236+
value: "letsencrypt-http01"
237+
# E2E_TEST_TOKEN — used by the e2e/staging contract tests
238+
# (`/api/v1/admin/e2e/*` routes) to bypass anonymous quotas. Marked
239+
# optional so a missing key doesn't keep the api from booting in a
240+
# fresh cluster.
241+
- name: E2E_TEST_TOKEN
242+
valueFrom:
243+
secretKeyRef:
244+
name: instant-secrets
245+
key: E2E_TEST_TOKEN
246+
optional: true
247+
# ── OAuth client IDs (PUBLIC — id, not the secret) ───────────────
248+
# The GITHUB_CLIENT_SECRET / GOOGLE_CLIENT_SECRET above are the
249+
# private halves; these are the public client IDs the frontend
250+
# uses to construct the OAuth-redirect URL.
251+
- name: GITHUB_CLIENT_ID
252+
valueFrom:
253+
secretKeyRef:
254+
name: instant-secrets
255+
key: GITHUB_CLIENT_ID
256+
optional: true
257+
- name: GOOGLE_CLIENT_ID
258+
valueFrom:
259+
secretKeyRef:
260+
name: instant-secrets
261+
key: GOOGLE_CLIENT_ID
262+
optional: true
263+
# ── Object-store backend (storage abstraction, 2026-05-20) ────────
264+
# OBJECT_STORE_BACKEND selects the impl: do-spaces (prod today),
265+
# r2, s3, or minio. The rest are backend-specific. See
266+
# common/storageprovider/factory.go.
267+
- name: OBJECT_STORE_BACKEND
268+
valueFrom:
269+
secretKeyRef:
270+
name: instant-secrets
271+
key: OBJECT_STORE_BACKEND
272+
optional: true
273+
- name: OBJECT_STORE_ENDPOINT
274+
valueFrom:
275+
secretKeyRef:
276+
name: instant-secrets
277+
key: OBJECT_STORE_ENDPOINT
278+
optional: true
279+
- name: OBJECT_STORE_REGION
280+
valueFrom:
281+
secretKeyRef:
282+
name: instant-secrets
283+
key: OBJECT_STORE_REGION
284+
optional: true
285+
- name: OBJECT_STORE_ACCESS_KEY
286+
valueFrom:
287+
secretKeyRef:
288+
name: instant-secrets
289+
key: OBJECT_STORE_ACCESS_KEY
290+
optional: true
291+
- name: OBJECT_STORE_SECRET_KEY
292+
valueFrom:
293+
secretKeyRef:
294+
name: instant-secrets
295+
key: OBJECT_STORE_SECRET_KEY
296+
optional: true
297+
- name: OBJECT_STORE_BUCKET
298+
valueFrom:
299+
secretKeyRef:
300+
name: instant-secrets
301+
key: OBJECT_STORE_BUCKET
302+
optional: true
303+
- name: OBJECT_STORE_SECURE
304+
valueFrom:
305+
secretKeyRef:
306+
name: instant-secrets
307+
key: OBJECT_STORE_SECURE
308+
optional: true
309+
- name: OBJECT_STORE_PUBLIC_URL
310+
valueFrom:
311+
secretKeyRef:
312+
name: instant-secrets
313+
key: OBJECT_STORE_PUBLIC_URL
314+
optional: true
195315
# ── Platform admin gates ─────────────────────────────────────────
196316
# Two independent gates protect /api/v1/<prefix>/customers/...:
197317
#
@@ -235,6 +355,36 @@ spec:
235355
# https://otlp.eu01.nr-data.net:4317 for EU accounts.
236356
- name: OTEL_EXPORTER_OTLP_ENDPOINT
237357
value: "https://otlp.nr-data.net:4317"
358+
# OTEL_EXPORTER_OTLP_HEADERS — the `api-key=<NR-license-key>` pair
359+
# the OTLP exporter sends to New Relic. Patched into the live
360+
# deployment on 2026-05-12 (B3 incident) so distributed tracing
361+
# works.
362+
#
363+
# CODIFIED INTENT: pull from secret (never commit the literal
364+
# license key). LIVE STATE: hardcoded as an inline `value:`
365+
# (history from the original kubectl patch).
366+
#
367+
# Reconciling these two requires the live deployment to FIRST be
368+
# patched to remove its inline value (so the strategic-merge
369+
# apply doesn't try to set both `value` and `valueFrom`):
370+
#
371+
# kubectl get secret instant-secrets -n instant -o json \
372+
# | jq -r '.data.OTEL_EXPORTER_OTLP_HEADERS' \
373+
# | base64 -d # confirm secret holds the right value first
374+
# kubectl patch deploy/instant-api -n instant --type=json \
375+
# -p='[{"op":"remove","path":"/spec/template/spec/containers/0/env/<INDEX>"}]'
376+
# # then `kubectl apply -f app.yaml` will re-add via secretKeyRef
377+
#
378+
# Until that one-time reconciliation runs, this entry stays
379+
# COMMENTED OUT — the dry-run conflict it produces is real and
380+
# documents this remaining drift. See APPLY-CHECKLIST.md.
381+
#
382+
# - name: OTEL_EXPORTER_OTLP_HEADERS
383+
# valueFrom:
384+
# secretKeyRef:
385+
# name: instant-secrets
386+
# key: OTEL_EXPORTER_OTLP_HEADERS
387+
# optional: true
238388
# VERSION is read by buildinfo at runtime (overrides the -ldflags
239389
# value baked at build time). Operators bump this per release;
240390
# alternatively, leave empty and rely on the image tag baked in

0 commit comments

Comments
 (0)