Skip to content

Commit e37248f

Browse files
ci: add manifest validation + manually-gated apply for infra/k8s
The infra repo had no CI on its k8s manifests — a typo or invalid schema only surfaced at hand-`kubectl apply` time against prod. Two workflows: * validate.yml — runs on every push/PR touching k8s/**: yamllint + kubeconform schema validation. Catches broken manifests pre-merge. * apply.yml — workflow_dispatch ONLY (the approval gate: a human must deliberately trigger it and type the APPLY confirm phrase; GitHub's native Environment "required reviewers" gate needs a paid plan the org is not on). Runs validate → `kubectl diff` (preview) → scoped `kubectl apply`. The apply DELIBERATELY excludes secret manifests (CHANGE_ME placeholders would clobber live prod secrets) and the three service Deployments (app/worker/provisioner — `:local` tags owned by the service auto-deploy pipelines). Needs the KUBECONFIG_B64 repo secret. Auto-apply-on-push was deliberately NOT used: the repo manifests are bootstrap templates that have drifted from live prod (placeholder secrets, `:local` image tags), so they are not yet a GitOps source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent e27394f commit e37248f

2 files changed

Lines changed: 237 additions & 0 deletions

File tree

.github/workflows/apply.yml

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
# infra — gated apply of k8s manifests to the prod cluster.
3+
#
4+
# APPROVAL MODEL: this workflow is `workflow_dispatch` ONLY — it never runs
5+
# on push. Applying to prod requires a human to deliberately open the
6+
# Actions tab, pick this workflow, and click "Run workflow", then type the
7+
# confirm phrase. That deliberate manual trigger IS the approval gate.
8+
# (GitHub's native Environment "required reviewers" gate needs a paid plan
9+
# the org is not on — `workflow_dispatch` is the free-plan equivalent and
10+
# matches the existing pin-prod-images.yml pattern.)
11+
#
12+
# SAFETY SCOPING — the apply DELIBERATELY excludes:
13+
# * secret manifests (secrets.yaml, infra-secrets.yaml, data/*secret*,
14+
# backups/instant-data-spaces-creds.yaml) — they carry CHANGE_ME
15+
# placeholders; applying them would clobber the live prod secrets.
16+
# * the 3 service Deployments (app.yaml, worker/deployment.yaml,
17+
# provisioner/deployment.yaml) — they hardcode `:local` image tags and
18+
# are owned by the api/worker/provisioner auto-deploy pipelines via
19+
# `kubectl set image`. Applying them would revert prod to a dead image.
20+
# * *-ha/values.yaml — Helm chart inputs, not k8s objects.
21+
# It runs `kubectl diff` first and prints it so the operator sees exactly
22+
# what will change before the apply step runs.
23+
#
24+
# Required repo secret: KUBECONFIG_B64 — base64 of a kubeconfig with apply
25+
# rights on the prod cluster (the same one the api repo uses; grab a fresh
26+
# static-token kubeconfig from the DigitalOcean control panel — the local
27+
# doctl-exec kubeconfig cannot be embedded).
28+
29+
name: apply
30+
31+
on:
32+
workflow_dispatch:
33+
inputs:
34+
confirm:
35+
description: 'Type APPLY to confirm applying infra/k8s to PROD.'
36+
required: true
37+
type: string
38+
39+
env:
40+
K8S_DIR: k8s
41+
42+
jobs:
43+
validate:
44+
name: Lint + schema-check before apply
45+
runs-on: ubuntu-latest
46+
steps:
47+
- uses: actions/checkout@v4
48+
- name: yamllint
49+
run: |
50+
python3 -m pip install --quiet yamllint
51+
cat > /tmp/yamllint.yml <<'YML'
52+
extends: relaxed
53+
rules:
54+
line-length: disable
55+
comments: disable
56+
comments-indentation: disable
57+
document-start: disable
58+
truthy: disable
59+
empty-lines: disable
60+
trailing-spaces: disable
61+
indentation: disable
62+
YML
63+
yamllint -c /tmp/yamllint.yml "${K8S_DIR}/"
64+
- name: kubeconform
65+
run: |
66+
VER=v0.6.7
67+
curl -sSL "https://github.com/yannh/kubeconform/releases/download/${VER}/kubeconform-linux-amd64.tar.gz" \
68+
| tar -xz -C /usr/local/bin kubeconform
69+
mapfile -t FILES < <(find "${K8S_DIR}" -type f \( -name '*.yaml' -o -name '*.yml' \) \
70+
! -path '*-ha/values.yaml' | sort)
71+
kubeconform -strict -ignore-missing-schemas -kubernetes-version 1.31.0 -summary "${FILES[@]}"
72+
73+
apply:
74+
name: Diff + apply to prod (manual gate)
75+
needs: validate
76+
runs-on: ubuntu-latest
77+
steps:
78+
- uses: actions/checkout@v4
79+
80+
- name: Confirm the APPLY phrase
81+
run: |
82+
if [ "${{ inputs.confirm }}" != "APPLY" ]; then
83+
echo "::error::confirm input was '${{ inputs.confirm }}', expected 'APPLY'. Aborting."
84+
exit 1
85+
fi
86+
echo "confirm phrase OK — proceeding."
87+
88+
- name: Set up kubectl
89+
uses: azure/setup-kubectl@v4
90+
with:
91+
version: 'latest'
92+
93+
- name: Configure kubeconfig from KUBECONFIG_B64
94+
env:
95+
KUBECONFIG_B64: ${{ secrets.KUBECONFIG_B64 }}
96+
run: |
97+
if [ -z "${KUBECONFIG_B64}" ]; then
98+
echo "::error::KUBECONFIG_B64 repo secret is not set on the infra repo."
99+
echo "Add it: a base64-encoded static-token kubeconfig for the prod cluster"
100+
echo "(the same value the api repo uses). Settings → Secrets → Actions."
101+
exit 1
102+
fi
103+
mkdir -p "$HOME/.kube"
104+
echo "$KUBECONFIG_B64" | base64 -d > "$HOME/.kube/config"
105+
chmod 600 "$HOME/.kube/config"
106+
kubectl config current-context
107+
108+
- name: Build the apply file list (scoped — excludes secrets + service Deployments)
109+
id: files
110+
run: |
111+
# Exclusions — see the header comment for the rationale.
112+
EXCLUDE=(
113+
"${K8S_DIR}/secrets.yaml"
114+
"${K8S_DIR}/infra-secrets.yaml"
115+
"${K8S_DIR}/data/minio-secret.yaml"
116+
"${K8S_DIR}/data/redis-provision-ha/secret.yaml"
117+
"${K8S_DIR}/backups/instant-data-spaces-creds.yaml"
118+
"${K8S_DIR}/app.yaml"
119+
"${K8S_DIR}/worker/deployment.yaml"
120+
"${K8S_DIR}/provisioner/deployment.yaml"
121+
)
122+
KEEP=()
123+
while IFS= read -r f; do
124+
skip=false
125+
for e in "${EXCLUDE[@]}"; do [ "$f" = "$e" ] && skip=true && break; done
126+
$skip || KEEP+=("$f")
127+
done < <(find "${K8S_DIR}" -type f \( -name '*.yaml' -o -name '*.yml' \) \
128+
! -path '*-ha/values.yaml' | sort)
129+
echo "Will apply ${#KEEP[@]} manifest(s); excluded ${#EXCLUDE[@]} (secrets + service Deployments)."
130+
printf '%s\n' "${KEEP[@]}" > /tmp/apply-files.txt
131+
printf ' apply: %s\n' "${KEEP[@]}"
132+
printf ' EXCLUDED: %s\n' "${EXCLUDE[@]}"
133+
134+
- name: kubectl diff (preview — what will change)
135+
run: |
136+
# diff exit code 1 just means "differences exist" — not an error.
137+
while IFS= read -r f; do
138+
echo "──── diff: $f ────"
139+
kubectl diff -f "$f" || true
140+
done < /tmp/apply-files.txt
141+
142+
- name: kubectl apply (scoped)
143+
run: |
144+
rc=0
145+
while IFS= read -r f; do
146+
echo "──── apply: $f ────"
147+
kubectl apply -f "$f" || rc=$?
148+
done < /tmp/apply-files.txt
149+
if [ "$rc" -ne 0 ]; then
150+
echo "::error::one or more manifests failed to apply (exit $rc)"
151+
exit "$rc"
152+
fi
153+
echo "all scoped manifests applied."

.github/workflows/validate.yml

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
# infra — manifest validation gate.
3+
#
4+
# Runs on every push and PR that touches k8s manifests. Catches broken
5+
# YAML and invalid Kubernetes object schemas BEFORE anyone `kubectl apply`s
6+
# them by hand — today a typo only surfaces at apply time against prod.
7+
#
8+
# This workflow deliberately does NOT `kubectl apply`. Auto-apply of this
9+
# repo to prod is unsafe in its current state:
10+
# * k8s/secrets.yaml / infra-secrets.yaml / data/*secret*.yaml carry
11+
# CHANGE_ME placeholders — applying them would clobber the live prod
12+
# secrets and take the whole platform down.
13+
# * k8s/app.yaml, worker/deployment.yaml, provisioner/deployment.yaml
14+
# hardcode `:local` image tags — applying them would revert all three
15+
# services to a non-existent image and undo their own auto-deploys.
16+
# A real auto-apply needs: (1) secrets moved out of git (sealed-secrets /
17+
# external-secrets), (2) image tags parameterised or those Deployments
18+
# owned solely by the service pipelines, (3) the manifests reconciled to
19+
# the actual live cluster state so apply is a no-op on a clean tree.
20+
# Until then, apply stays a deliberate manual step. See CLAUDE.md rule 15.
21+
22+
name: validate
23+
24+
on:
25+
push:
26+
branches: [master]
27+
paths: ['k8s/**', '.github/workflows/validate.yml']
28+
pull_request:
29+
paths: ['k8s/**', '.github/workflows/validate.yml']
30+
workflow_dispatch:
31+
32+
jobs:
33+
validate:
34+
name: Lint + schema-check k8s manifests
35+
runs-on: ubuntu-latest
36+
steps:
37+
- name: Checkout
38+
uses: actions/checkout@v4
39+
40+
- name: yamllint
41+
run: |
42+
python3 -m pip install --quiet yamllint
43+
# Relaxed: k8s manifests routinely have long lines and varied
44+
# indent; we care about genuine syntax errors, not style.
45+
cat > /tmp/yamllint.yml <<'YML'
46+
extends: relaxed
47+
rules:
48+
line-length: disable
49+
comments: disable
50+
comments-indentation: disable
51+
document-start: disable
52+
truthy: disable
53+
empty-lines: disable
54+
trailing-spaces: disable
55+
indentation: disable
56+
YML
57+
yamllint -c /tmp/yamllint.yml k8s/
58+
59+
- name: Install kubeconform
60+
run: |
61+
VER=v0.6.7
62+
curl -sSL "https://github.com/yannh/kubeconform/releases/download/${VER}/kubeconform-linux-amd64.tar.gz" \
63+
| tar -xz -C /usr/local/bin kubeconform
64+
kubeconform -v
65+
66+
- name: kubeconform schema validation
67+
run: |
68+
set -euo pipefail
69+
# Exclude Helm values files (k8s/data/*-ha/values.yaml) — those
70+
# are Helm chart inputs, not Kubernetes objects, so they have no
71+
# k8s schema to check against.
72+
# -ignore-missing-schemas: CRDs (cert-manager, Cilium, CNPG)
73+
# have no built-in schema; skip them rather than fail.
74+
# -strict: reject unknown/duplicate fields in core objects.
75+
mapfile -t FILES < <(find k8s -type f \( -name '*.yaml' -o -name '*.yml' \) \
76+
! -path '*-ha/values.yaml' | sort)
77+
echo "validating ${#FILES[@]} manifest(s):"
78+
printf ' %s\n' "${FILES[@]}"
79+
kubeconform \
80+
-strict \
81+
-ignore-missing-schemas \
82+
-kubernetes-version 1.31.0 \
83+
-summary \
84+
"${FILES[@]}"

0 commit comments

Comments
 (0)