Wave 7: Argo Rollouts canary + metric-gated auto-rollback for instant-api (operator-apply)

[mastermanas805]

Wave 7 — canary deploy with metric-gated AUTO-ROLLBACK for instant-api

A bad instant-api image today reaches 100% of api.instanode.dev traffic the moment kubectl set image deployment/instant-api runs (RollingUpdate), and a regression is only caught after users hit it. This ships the operator-apply manifests + runbook to make a bad deploy auto-roll-back on metric regression instead of being detected after the fact.

Zero prod risk — operator-apply only

• Nothing here touches prod. infra has no auto-apply (CLAUDE.md rule 15).
• The live deploy path is UNCHANGED: api/.github/workflows/deploy.yml → kubectl set image deployment/instant-api and k8s/app.yaml (the live Deployment) are not modified. No CI workflow is changed.
• Migration is a deliberate, fully reversible operator action documented in the runbook.

The canary ladder

setWeight 10 → pause 5m (bake) → ANALYSE → setWeight 50 → pause 5m → ANALYSE → setWeight 100 → pause 2m → promote
                                    │                                  │
                                    └── metric regresses ──────────────┴──► ABORT → 100% traffic back to STABLE (auto-rollback)

What the AnalysisTemplate gates on (real prod metrics, names verified vs prometheus-rules.yaml + newrelic/alerts/)

• (a) HTTP 5xx rate — sum(rate(instant_http_errors_total{job="instant-api",status_class="5xx"}[2m])) / clamp_min(sum(rate(instant_http_request_duration_seconds_count{job="instant-api"}[2m])),1e-9) < 0.02
• (b) deploy/health — sum(increase(instant_deploy_job_failed_detected_total[5m])) == 0 and sum(up{job="instant-api"}) >= 1
• (c) auth-probe (AUTH-004, CLAUDE.md rule 26) — sum(increase(instant_auth_probe_outcome_total{result="fail"}[10m])) == 0

A breach of failureLimit fails the AnalysisRun → Rollout aborts → traffic returns to the stable ReplicaSet. Prometheus provider is the default; New Relic provider blocks are included (commented) because prod observability is NR today (no in-cluster Prometheus).

Files

File	What
k8s/canary/rollout-instant-api.yaml	Rollout/instant-api canary, workloadRef to the existing Deployment → drop-in parity (replicas/resources/probes/env/imagePullSecrets/affinity), zero pod-spec duplication
k8s/canary/analysistemplate-instant-api.yaml	the metric gate (5xx / deploy-job-failures / api-up / auth-probe), failureLimit/inconclusiveLimit
k8s/canary/services-instant-api-canary.yaml	instant-api-stable + instant-api-canary Services (existing Service + api Ingress untouched)
CANARY-ROLLOUT-RUNBOOK.md	install controller → provider → safe workloadRef cutover → CI flip (operator, NOT this PR) → manual abort/undo → full back-out

Operator action required (NOT done in this PR)

• Argo Rollouts is NOT installed (kubectl get crd rollouts.argoproj.io → NotFound in do-nyc3-instant-prod) → runbook §install.
• No in-cluster Prometheus today (observability is NR) → runbook §provider (stand up Prom scraping job="instant-api", OR switch the template to the NR provider).
• The api deploy.yml flip to kubectl argo rollouts set image is documented for the operator to do at migration time so current deploys never break.

Rule 25 — canary observability shipped in-PR

• NR alert newrelic/alerts/canary-analysis-aborted.json (auto-pickup by newrelic/apply.sh)
• 2 tiles on newrelic/dashboards/instanode-reliability.json (rollout phase + AnalysisRun outcomes)
• observability/METRICS-CATALOG.md row (rollout_info / analysis_run_info, lazy until controller installed + scraped)

validate.yml stays green

kubeconform -ignore-missing-schemas skips the Argo Rollouts CRDs (Rollout/AnalysisTemplate) — exactly like the existing cert-manager/Cilium/CNPG CRDs — while the two Services validate strictly. Verified locally with the exact CI command: 99 valid / 0 invalid / 9 skipped, yamllint clean on k8s/canary/.

🤖 Generated with Claude Code