incident: production 500s after PR #72 deploy (missing JWT config)

[chrisjwalk]

Root Cause Analysis — 2026-03-16

Summary

After merging PR #72 (JWT + HttpOnly cookie migration), the production site returned HTTP 500 on all auth-related endpoints. The root cause was three missing Azure App Service environment variables that were introduced by the PR but never provisioned in the production environment.

Timeline

Time (UTC)	Event
~22:05	PR #72 merged; deploy workflow triggered automatically
~22:06	Deploy succeeded — Azure App Service restarted with new binary
~01:09 next day	User reported 500 errors on the backend
~01:13	Investigation confirmed JWT_KEY, Jwt__Issuer, Jwt__Audience missing from App Service config
~01:14	All three env vars set via az webapp config appsettings set; app restarted
~01:15	Production confirmed healthy — /api/weatherforecasts 200, /api/auth/refresh 401

Root Cause

PR #72 added AddJwtBearer authentication which reads signing key and audience/issuer from config:

• Dev: appsettings.Development.json (present, committed)
• Prod: environment variables JWT_KEY, Jwt__Issuer, Jwt__Audience (not set in Azure)

Without JWT_KEY, token generation/validation threw ArgumentNullException at runtime. The app started successfully (startup code did not validate required config) so the deploy appeared to succeed, masking the problem until a user hit an auth endpoint.

Contributing Factors

1. No startup config validation — missing required values were not caught at boot
2. No post-deploy smoke test — the deploy pipeline reported success without verifying the app was actually responding correctly
3. No runbook / deployment checklist documenting required environment variables

Impact

• Scope: All auth endpoints (/api/auth/login, /api/auth/refresh, /api/auth/logout) returning 500. Non-auth endpoints unaffected.
• Duration: ~3 hours (22:05 UTC → 01:15 UTC)
• Severity: High — users unable to log in on production
• Data loss: None

Remediation (immediate)

Set missing env vars via Azure CLI:

az webapp config appsettings set \
  --name AngularCliNetcoreNgrxStarter \
  --resource-group AngularCliNetcoreNgrxStarter \
  --settings JWT_KEY=<secret> Jwt__Issuer=<url> Jwt__Audience=<url>

Preventive Actions

Tracked in issue #75:

1. Fail-fast startup validation in Program.cs — app refuses to start if required config is absent; Azure will surface the crash in logs immediately
2. Post-deploy smoke test in deploy.yml — curl /api/weatherforecasts after deploy; fail the workflow on 5xx so broken deploys are caught in CI before the user ever sees them

[chrisjwalk-bot]

RCA documented in this issue. Preventive measures shipped in #77 (fail-fast startup validation, /health/live + /health/ready endpoints, post-deploy smoke test). Closing.