Update docs and journal for Day 3 progress

- CLAUDE.md: reflect completed stages 5-9 and new file locations
- architecture.md: add deploy.yml and docker.yml to CI/CD section
- implementation-plan.md: update progress dashboard with current status
- journal: add Day 3 entry covering all infrastructure work

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: johnmathews

CLAUDE.md

@@ -22,22 +22,32 @@ loan documents.
 - **Database:** PostgreSQL with pgvector (metadata + embeddings)
 - **Blob storage:** Azure Blob Storage (original PDFs, optional)
 
-### Planned (not yet implemented)
-- **Autoscaling:** KEDA (queue-depth based)
-- **Monitoring:** Prometheus + Grafana
-- **Chaos engineering:** Chaos Mesh
+### Implemented (Day 3)
+- **Autoscaling:** KEDA ScaledObjects (queue-depth based, YAMLs in k8s/scaling/)
+- **Monitoring:** Prometheus + Grafana dashboard (grafana/documentstream-dashboard.json)
+- **Chaos engineering:** Chaos Mesh experiments (k8s/chaos/)
+- **Load testing:** Locust (locust/locustfile.py)
+- **CI/CD:** GitHub Actions deploy workflow (.github/workflows/deploy.yml)
+
+### Not yet done (needs live AKS cluster)
+- Azure infra provisioning (scripts ready in infra/)
+- Build/push images to ACR and deploy to AKS
+- Import Grafana dashboard, apply KEDA/Chaos manifests
+- End-to-end demo rehearsal
 
 ## Project Structure
 - `src/gateway/` — FastAPI API + web UI (dual-mode: sync or async via Redis)
 - `src/worker/` — Extract, classify, semantic, store modules + Redis queue + worker runners
 - `src/generator/` — PDF document generator (5 templates, CLI tool)
 - `demo_samples/` — One complete loan scenario (5 PDFs, committed to git for visibility)
 - `tests/` — All tests (83 tests)
-- `k8s/` — Kubernetes manifests (empty — Day 2)
-- `infra/` — Azure setup/teardown scripts (empty — Day 2)
-- `locust/` — Load testing (empty — Day 3)
-- `grafana/` — Dashboard JSON (empty — Day 2)
-- `docs/` — Documentation (architecture, classification, demo guide, dictionary)
+- `k8s/base/` — Kubernetes base manifests (9 files: namespace, configmap, deployments, service, ingress, kustomization)
+- `k8s/scaling/` — KEDA ScaledObjects for extract, classify, store workers
+- `k8s/chaos/` — Chaos Mesh experiments (pod-kill, network-delay, cpu-stress)
+- `infra/` — Azure setup/teardown/helm-install scripts
+- `locust/` — Locust load test (locustfile.py)
+- `grafana/` — Grafana dashboard JSON (7 panels)
+- `docs/` — Documentation (architecture, classification, demo guide, dictionary, implementation plan)
 - `journal/` — Development journal
 
 ## Commands
@@ -59,7 +69,8 @@ loan documents.
 - Workers use Redis consumer groups for at-least-once delivery
 - SIGTERM graceful shutdown on all workers (finish current message before exiting)
 
-### Target architecture (Day 2-3)
-- Each pipeline stage as a separate K8s Deployment scaled by KEDA
+### Architecture
+- Each pipeline stage is a separate K8s Deployment scaled by KEDA
 - Documents flow through Redis Streams: raw-docs → extracted → classified → stored
 - Store worker persists to PostgreSQL (pgvector) + Azure Blob Storage
+- CI/CD: ci.yml (lint+test), docker.yml (ghcr.io push), deploy.yml (ACR build + AKS deploy)

docs/architecture.md

@@ -217,7 +217,8 @@ When stopped (`az aks stop` + `az postgres flexible-server stop`): ~€0.01/hr (
 
 GitHub Actions workflows:
 - **ci.yml** — On push to main and PRs: ruff lint, ruff format check, pytest with coverage
-- **deploy.yml** — *Not yet created.* Will build images → push to ACR → deploy to AKS
+- **docker.yml** — On push to main: build and push images to ghcr.io/johnmathews/k8s
+- **deploy.yml** — On push to main (src/ or k8s/ changes): build → push to ACR → deploy to AKS
 
 ---
 

docs/implementation-plan.md

@@ -2,7 +2,7 @@
 
 **Timeline:** 3 days (2026-03-28 to 2026-03-30)
 **Interview:** After Day 3
-**Last updated:** 2026-03-29
+**Last updated:** 2026-03-30
 
 ---
 
@@ -12,16 +12,16 @@
 |---|---|---|---|---|
 | -- | **Day 1: Foundation** | -- | -- | **DONE** |
 | 0 | Tool setup (helm, kustomize) | MUST | 15min | DONE |
-| 1 | Azure infrastructure | MUST | 1.5-2h | PARTIAL |
+| 1 | Azure infrastructure | MUST | 1.5-2h | PARTIAL (scripts written, not executed) |
 | 2 | Redis Streams pipeline refactor | MUST | 2.5-3h | DONE |
 | 3 | K8s manifests | MUST | 2-2.5h | DONE |
-| 4 | Build, push, deploy to AKS | MUST | 1-1.5h | TODO |
-| 5 | KEDA autoscaling | MUST | 1-1.5h | TODO |
-| 6 | Grafana dashboard | HIGH | 1.5-2h | TODO |
-| 7 | Chaos Mesh experiments | MEDIUM | 1h | TODO |
-| 8 | Locust load testing | MEDIUM | 1h | TODO |
-| 9 | CI/CD deploy workflow | MEDIUM | 1h | TODO |
-| 10 | Rolling update demo prep | LOW | 30min | TODO |
+| 4 | Build, push, deploy to AKS | MUST | 1-1.5h | TODO (needs Azure) |
+| 5 | KEDA autoscaling | MUST | 1-1.5h | DONE (YAMLs written, needs apply) |
+| 6 | Grafana dashboard | HIGH | 1.5-2h | DONE (JSON written, needs import) |
+| 7 | Chaos Mesh experiments | MEDIUM | 1h | DONE (YAMLs written, needs apply) |
+| 8 | Locust load testing | MEDIUM | 1h | DONE (locustfile written, needs run) |
+| 9 | CI/CD deploy workflow | MEDIUM | 1h | DONE |
+| 10 | Rolling update demo prep | LOW | 30min | TODO (live demo technique) |
 | 11 | Polish and demo rehearsal | MUST | 1.5-2h | TODO |
 
 **If time runs short:** Cut from the bottom. Stages 0-5 + 11 are non-negotiable. Stage 6 (Grafana) is

journal/260330-day3-keda-grafana-chaos-locust-cicd.md

@@ -0,0 +1,59 @@
+# Day 3: KEDA, Grafana, Chaos Mesh, Locust, and CI/CD Deploy
+
+**Date:** 2026-03-30
+
+## What Was Done
+
+Completed Stages 5-9 of the implementation plan. All files are written and ready to
+apply once the AKS cluster is provisioned. No new Python application code — this was
+entirely infrastructure/config work.
+
+### Stage 5: KEDA ScaledObjects
+- Created 3 ScaledObject manifests in `k8s/scaling/`
+- Each targets a worker deployment and watches its corresponding Redis stream
+- Config: pollingInterval 15s, cooldownPeriod 60s, min 1 / max 8 replicas, lagCount threshold 5
+- Uses the `redis-streams` trigger type pointing at `redis-master.documentstream.svc.cluster.local:6379`
+
+### Stage 6: Grafana Dashboard
+- Created `grafana/documentstream-dashboard.json` with 7 panels
+- Row 1: Pod count (bar gauge), Redis queue depth (stat), Pod restarts (stat)
+- Row 2: CPU usage per pod (timeseries), Memory usage per pod (timeseries)
+- Row 3: KEDA scaling metrics (timeseries), Network I/O (timeseries)
+- 5-second auto-refresh for live demo, color thresholds on stat panels
+
+### Stage 7: Chaos Mesh Experiments
+- `k8s/chaos/pod-kill.yaml` — Kills 2 classify-worker pods (demonstrates self-healing)
+- `k8s/chaos/network-delay.yaml` — 500ms latency on store-workers (demonstrates resilience)
+- `k8s/chaos/cpu-stress.yaml` — 80% CPU burn on classify-workers (demonstrates KEDA scale-up)
+- All include descriptive comments and usage commands
+
+### Stage 8: Locust Load Test
+- `locust/locustfile.py` with 4 weighted tasks
+- Upload PDF (weight 3), generate scenario (weight 1), list docs (weight 5), health check (weight 2)
+- PDF generated once at class level, reused per request
+- Supports local, AKS, and headless (CI) modes
+
+### Stage 9: CI/CD Deploy Workflow
+- `.github/workflows/deploy.yml` — triggers on push to main when src/ or k8s/ files change
+- Builds both gateway and worker images to ACR with SHA tags
+- Applies K8s manifests and waits for rollout status
+- Added `workflow_call` trigger to `ci.yml` so deploy.yml can reuse it as a gate
+
+### Documentation Updates
+- Updated CLAUDE.md to reflect all new files and current project state
+- Updated architecture.md with deploy.yml reference
+- Updated implementation-plan.md progress dashboard
+
+## What's Left
+
+All remaining work requires a live AKS cluster:
+1. Run `infra/setup.sh` to provision Azure resources
+2. Run `infra/helm-install.sh` to install Helm charts
+3. Build and push images to ACR
+4. `kubectl apply -k k8s/base/` + scaling + chaos manifests
+5. Import Grafana dashboard
+6. End-to-end demo rehearsal
+
+## Test Status
+- 83 tests passing, 88% coverage
+- No new tests needed (all new files are YAML/JSON/config)