Update docs for blob storage, Prometheus metrics, and new K8s concepts

- Dictionary: add ServiceMonitor, kubectl rollout restart, pod READY column,
  kubectl get all limitations, default namespace, Azurite, ACR image building
- Architecture: update store worker details, gateway /metrics endpoint, mark
  Helm charts as installed
- Demo guide: mention blob uploads and Prometheus metrics in talking points

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: johnmathews

docs/architecture.md

@@ -135,7 +135,7 @@ storing → completed).
 | Aspect | Detail |
 |---|---|
 | Framework | FastAPI |
-| Endpoints | `/health`, `/api/documents`, `/api/generate`, `/` (web UI) |
+| Endpoints | `/health`, `/api/documents`, `/api/generate`, `/metrics`, `/` (web UI) |
 | K8s role | Single Deployment with a Service and Ingress |
 | Scaling | HPA based on CPU (not queue-based — it handles HTTP, not queue work) |
 
@@ -165,9 +165,10 @@ storing → completed).
 | Aspect | Detail |
 |---|---|
 | Database | PostgreSQL with pgvector extension |
-| Blob store | Azure Blob Storage (optional, via `BLOB_CONNECTION_STRING`) |
+| Blob store | Azure Blob Storage (via `BLOB_CONNECTION_STRING` in K8s Secret) |
+| Metrics | Prometheus counters: `documentstream_blob_uploads_total`, `documentstream_blob_bytes_total` (by doc_type) |
 | Input | Redis stream `classified` |
-| Stored data | Document metadata, classification results, vector embedding (384 dims), blob URL |
+| Stored data | Document metadata, doc_type, classification results, vector embedding (384 dims), blob URL |
 | Scaling | KEDA ScaledObject watching `classified` stream depth |
 
 ### Queue Module (`src/worker/queue.py`)
@@ -211,7 +212,7 @@ When stopped (`az aks stop` + `az postgres flexible-server stop`): ~€0.01/hr (
 | chaos-mesh/chaos-mesh | chaos-mesh | Chaos engineering |
 | ingress-nginx | ingress-nginx | HTTP routing |
 
-*Not yet installed — these are deployed when the AKS cluster is provisioned.*
+All charts are installed and running on the live AKS cluster.
 
 ### CI/CD
 

docs/demo-guide.md

@@ -36,7 +36,9 @@ Before the interview:
 **Show:** Click "Generate" with 5 scenarios
 
 > "Each loan scenario generates 5 linked documents. They share the same loan ID, client,
-> and property — just like a real bank's document management system."
+> and property — just like a real bank's document management system. Each PDF is uploaded
+> to Azure Blob Storage automatically — you can see the count and total size per document
+> type on the Grafana dashboard."
 
 **Point out the two classifier columns:**
 > "Every document goes through two classifiers. The rule-based classifier handles privacy
@@ -148,7 +150,9 @@ Before the interview:
 > "The full pipeline: FastAPI gateway receives uploads, puts them on a Redis stream.
 > Extract workers pull text with PyMuPDF. Classify workers run both rule-based and
 > semantic classification. Store workers save metadata and embeddings to PostgreSQL
-> with pgvector, and original PDFs to Azure Blob Storage."
+> with pgvector, and original PDFs to Azure Blob Storage. The gateway exposes Prometheus
+> metrics — blob upload counts and sizes by document type — which Prometheus scrapes
+> via a ServiceMonitor and Grafana displays in real time."
 
 **Show:** Cost breakdown
 

docs/dictionary.md

@@ -38,7 +38,34 @@ file, but managed by K8s and versioned.
 
 ### Secret
 Same as ConfigMap but for sensitive data (passwords, API keys). Base64-encoded at rest.
-Like Docker secrets.
+Like Docker secrets. Referenced via `secretRef` in a Deployment's `envFrom` — if listed
+after a ConfigMap, Secret values override ConfigMap values with the same key.
+
+**Important:** Never commit Secrets to git. Create them via `kubectl create secret` or
+apply a gitignored YAML file. GitHub Push Protection will block pushes containing keys.
+
+### ServiceMonitor
+A Custom Resource (CRD) used by the **kube-prometheus-stack** to tell Prometheus which
+services to scrape. Pod annotations like `prometheus.io/scrape: "true"` do **not** work
+with kube-prometheus-stack — you must create a ServiceMonitor instead.
+
+Key gotcha: the ServiceMonitor needs a `release: prometheus` label (or whatever label
+selector your Prometheus instance uses). Without it, Prometheus silently ignores it.
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  labels:
+    release: prometheus  # required!
+spec:
+  selector:
+    matchLabels:
+      app: gateway
+  endpoints:
+    - port: http         # must match a named port on the Service
+      path: /metrics
+```
 
 ---
 
@@ -57,6 +84,17 @@ How K8s deploys a new version: start new pods, wait until they're ready, then ki
 At no point are zero pods running. This gives you **zero-downtime deployments**. If the new
 pods fail their readiness probes, K8s stops the rollout automatically.
 
+### `kubectl rollout restart`
+Triggers a rolling restart of all pods in a deployment without changing any YAML. Needed
+when a ConfigMap or Secret changes — K8s does **not** automatically restart pods when their
+ConfigMap values change. You have to tell it to.
+
+```bash
+kubectl rollout restart deployment/gateway deployment/store-worker
+kubectl rollout status deployment/gateway   # watch progress
+kubectl rollout history deployment/gateway  # see past revisions
+```
+
 ### Rollback
 Undo a deployment: `kubectl rollout undo deployment/my-app`. K8s keeps the previous pod
 spec and can revert to it instantly.
@@ -67,6 +105,28 @@ spec and can revert to it instantly.
 - **Limit:** "This container must never use more than 512MB RAM."
   If it exceeds the memory limit, K8s kills it (OOMKilled).
 
+### Pod READY Column
+When you run `kubectl get pods`, the READY column shows `1/1` or `2/2`. This is
+`READY_CONTAINERS/TOTAL_CONTAINERS` in that pod. A pod with an app + sidecar container
+would show `2/2` when both are ready, or `1/2` if one is still starting.
+
+### `kubectl get all` Limitations
+Despite the name, `kubectl get all` only returns a hardcoded subset: Pods, Services,
+Deployments, ReplicaSets, StatefulSets, DaemonSets, Jobs, CronJobs. It does **not**
+include HPAs, ConfigMaps, Secrets, Ingresses, PVCs, ServiceMonitors, ScaledObjects, etc.
+
+To see everything in a namespace, be explicit:
+```bash
+kubectl get deploy,svc,hpa,ingress,configmap,scaledobject -n documentstream
+```
+
+### Default Namespace
+Set a default namespace to avoid typing `-n documentstream` on every command:
+```bash
+kubectl config set-context --current --namespace=documentstream
+kubectl config view --minify | grep namespace  # verify
+```
+
 ### Horizontal Pod Autoscaler (HPA)
 Watches a metric (CPU, memory, custom) and adjusts the number of pod replicas. Example:
 "If average CPU > 70%, add more pods. If < 30%, remove pods." Like auto-scaling in cloud
@@ -154,6 +214,27 @@ not in use (you only pay for disk storage while stopped).
 
 ### Azure Blob Storage
 Object storage for files (like S3). We store the original PDF files here. Extremely cheap.
+PDFs are organized as `{doc_id}/{loan_id}/{doc_type}.pdf` in a container called `documents`.
+
+### Azurite
+Microsoft's local emulator for Azure Storage. Runs as a Docker container and provides the
+same API as real Azure Blob Storage. Used in `docker-compose.yml` for local dev so you don't
+need a real Azure account to test blob uploads.
+
+### Building and Pushing Images to ACR
+`docker build` only creates the image **locally**. To deploy to AKS, you must also push:
+
+```bash
+# Option 1: Build locally + push (need --platform for ARM Mac → AMD64 cluster)
+docker build --platform linux/amd64 -t acrdocumentstream.azurecr.io/gateway:latest -f src/gateway/Dockerfile .
+az acr login --name acrdocumentstream
+docker push acrdocumentstream.azurecr.io/gateway:latest
+
+# Option 2: Build remotely on ACR (no platform flag needed, builds on Linux)
+az acr build --registry acrdocumentstream --image gateway:latest --file src/gateway/Dockerfile .
+```
+
+After pushing, `kubectl rollout restart deployment/gateway` tells K8s to pull the new image.
 
 ---