|
| 1 | +# Agentic DevOps Reference App |
| 2 | + |
| 3 | +A purpose-built microservices application used as the hands-on lab target for the **Agentic DevOps** course. It gives participants a realistic multi-service system to practice AI-assisted operations: health monitoring, log triage, database investigation, deployment automation, and observability. |
| 4 | + |
| 5 | +## What it is |
| 6 | + |
| 7 | +The app simulates a small production-grade platform with three backend services, a database, and a live health dashboard. It is intentionally observable — every service exposes structured health endpoints, version metadata, and generates write traffic — so that AI agents have something meaningful to query and reason about. |
| 8 | + |
| 9 | +## Architecture |
| 10 | + |
| 11 | +``` |
| 12 | +Browser |
| 13 | + └── Dashboard (SvelteKit → nginx :3000) |
| 14 | + └── /api-gateway/* ──proxy──► API Gateway (:8080) |
| 15 | + └── /catalog/* ──proxy──► Catalog (:8081) |
| 16 | + └── /worker/* ──proxy──► Worker (:8082) |
| 17 | +
|
| 18 | +API Gateway (:8080) |
| 19 | + ├── GET /api/status (aggregates catalog + worker health) |
| 20 | + ├── GET /version |
| 21 | + └── GET /health/live|ready |
| 22 | +
|
| 23 | +Catalog (:8081) Worker (:8082) |
| 24 | + ├── GET /items ├── POST /events |
| 25 | + ├── GET /items/:id ├── GET /events/recent |
| 26 | + ├── GET /version ├── GET /version |
| 27 | + └── GET /health/live|ready └── GET /health/live|ready |
| 28 | + │ │ |
| 29 | + └──────────┬───────────────────┘ |
| 30 | + ▼ |
| 31 | + PostgreSQL :5432 |
| 32 | + ├── items (catalog reads) |
| 33 | + └── events (worker writes, heartbeat every 60s) |
| 34 | +``` |
| 35 | + |
| 36 | +### Services |
| 37 | + |
| 38 | +| Service | Port | Language | Role | |
| 39 | +|---------|------|----------|------| |
| 40 | +| dashboard | 3000 | SvelteKit + nginx | Health dashboard UI, proxies API calls to backends | |
| 41 | +| api-gateway | 8080 | Rust / Axum | Aggregates catalog + worker health for the dashboard | |
| 42 | +| catalog | 8081 | Rust / Axum | Read-only item catalog backed by PostgreSQL | |
| 43 | +| worker | 8082 | Rust / Axum | Event writer; background heartbeat loop every 60s | |
| 44 | +| postgres | 5432 | PostgreSQL 16 | Single database with `items` and `events` tables | |
| 45 | + |
| 46 | +### Key Design Choices |
| 47 | + |
| 48 | +- **Rust workspace** — all three backend services share a `services/shared` crate that injects `VERSION` and `GIT_SHA` at compile time. |
| 49 | +- **Health endpoints** — every service has `/health/live` (process alive) and `/health/ready` (DB connection available). The dashboard polls these every 30 s. |
| 50 | +- **Degraded state** — when a service is unreachable the dashboard shows "last known state" rather than crashing. This is intentional for observability labs. |
| 51 | +- **Observable write traffic** — the worker writes a heartbeat event every 60 s, giving agents a live data stream to query. |
| 52 | +- **Seed data** — the DB migration pre-populates `items` with realistic-looking service names (including one intentionally `degraded`) for agent query labs. |
| 53 | + |
| 54 | +## Prerequisites |
| 55 | + |
| 56 | +- Docker (with Compose V2) — required for both deployment modes |
| 57 | +- `make` — orchestrates all commands |
| 58 | +- `kind` + `kubectl` + `helm` — required only for the Kubernetes deployment |
| 59 | + |
| 60 | +## Deploy with Docker Compose |
| 61 | + |
| 62 | +The quickest path. No Kubernetes required. |
| 63 | + |
| 64 | +```bash |
| 65 | +# Build all images and start the stack |
| 66 | +make compose-up |
| 67 | + |
| 68 | +# Tail logs across all services |
| 69 | +make compose-logs |
| 70 | + |
| 71 | +# Tear down (removes containers; PostgreSQL data persists in named volume) |
| 72 | +make compose-down |
| 73 | +``` |
| 74 | + |
| 75 | +Once running: |
| 76 | + |
| 77 | +| URL | What you see | |
| 78 | +|-----|-------------| |
| 79 | +| http://localhost:3000 | Health dashboard | |
| 80 | +| http://localhost:8080 | API Gateway (JSON index + endpoints) | |
| 81 | +| http://localhost:8081 | Catalog service | |
| 82 | +| http://localhost:8082 | Worker service | |
| 83 | +| localhost:5432 | PostgreSQL (`refapp` / `refapp-lab-password` / `refapp`) | |
| 84 | + |
| 85 | +### How it starts up |
| 86 | + |
| 87 | +``` |
| 88 | +postgres ──healthy──► catalog ──┐ |
| 89 | + ├──► api-gateway ──► dashboard |
| 90 | +postgres ──healthy──► worker ──┘ |
| 91 | +``` |
| 92 | + |
| 93 | +Docker Compose respects `depends_on` health conditions. PostgreSQL must pass its `pg_isready` check before catalog and worker start. The DB migration (`001_init.sql`) runs automatically via `docker-entrypoint-initdb.d`. |
| 94 | + |
| 95 | +### Rebuilding after code changes |
| 96 | + |
| 97 | +```bash |
| 98 | +# Rebuild a single service (e.g. after editing catalog) |
| 99 | +docker compose build catalog |
| 100 | +docker compose up -d catalog |
| 101 | + |
| 102 | +# Force full rebuild (bypasses layer cache) |
| 103 | +docker compose build --no-cache |
| 104 | +docker compose up -d |
| 105 | +``` |
| 106 | + |
| 107 | +## Deploy on Kubernetes (KIND) |
| 108 | + |
| 109 | +Uses a local KIND cluster with Helm. Includes Prometheus + Grafana for the observability labs. |
| 110 | + |
| 111 | +### One-command deploy |
| 112 | + |
| 113 | +```bash |
| 114 | +make deploy |
| 115 | +``` |
| 116 | + |
| 117 | +This runs the full sequence: |
| 118 | + |
| 119 | +1. `cluster` — creates a KIND cluster named `lab` (idempotent; skips if already exists) |
| 120 | +2. `db` — installs PostgreSQL via Bitnami Helm chart into the `db` namespace |
| 121 | +3. `monitoring` — installs `kube-prometheus-stack` into the `monitoring` namespace |
| 122 | +4. `build` — builds all four Docker images locally |
| 123 | +5. `load-images` — loads images into the KIND cluster (bypasses a registry) |
| 124 | +6. `app` — installs the `helm/reference-app` chart into the `app` namespace |
| 125 | + |
| 126 | +Once deployed: |
| 127 | + |
| 128 | +| URL | What you see | |
| 129 | +|-----|-------------| |
| 130 | +| http://localhost:30080 | Health dashboard | |
| 131 | +| http://localhost:30090 | Grafana (admin / admin) | |
| 132 | +| http://localhost:30091 | Prometheus | |
| 133 | + |
| 134 | +### Individual steps |
| 135 | + |
| 136 | +```bash |
| 137 | +make cluster # Create KIND cluster only |
| 138 | +make db # Install PostgreSQL |
| 139 | +make monitoring # Install Prometheus + Grafana |
| 140 | +make build # Build Docker images |
| 141 | +make load-images # Load images into KIND |
| 142 | +make app # Deploy the application chart |
| 143 | +make status # Show pod status across all namespaces |
| 144 | +make destroy # Delete the entire KIND cluster |
| 145 | +``` |
| 146 | + |
| 147 | +### KIND cluster config |
| 148 | + |
| 149 | +The cluster config is at `../infrastructure/kind/cluster-config.yaml` (relative to this directory). It sets up the node port mappings that expose the dashboard, Grafana, and Prometheus on localhost. |
| 150 | + |
| 151 | +### Helm chart |
| 152 | + |
| 153 | +The chart at `helm/reference-app/` deploys all four application services as Kubernetes Deployments with Services. Key values in `helm/reference-app/values.yaml`: |
| 154 | + |
| 155 | +- Image tags default to `1.0.0` (matches the `IMAGE_TAG` in the Makefile) |
| 156 | +- Services are exposed via NodePort for local KIND access |
| 157 | +- `DATABASE_URL`, `CATALOG_URL`, and `WORKER_URL` are injected as environment variables |
| 158 | + |
| 159 | +## Project structure |
| 160 | + |
| 161 | +``` |
| 162 | +reference-app/ |
| 163 | +├── Cargo.toml # Rust workspace root |
| 164 | +├── docker-compose.yml # Docker Compose stack |
| 165 | +├── Makefile # All deployment commands |
| 166 | +├── services/ |
| 167 | +│ ├── shared/ # Shared crate: VERSION + GIT_SHA constants |
| 168 | +│ ├── api-gateway/ # Rust/Axum service, port 8080 |
| 169 | +│ │ ├── src/main.rs |
| 170 | +│ │ └── Dockerfile |
| 171 | +│ ├── catalog/ # Rust/Axum service, port 8081 |
| 172 | +│ │ ├── src/main.rs |
| 173 | +│ │ ├── migrations/001_init.sql |
| 174 | +│ │ └── Dockerfile |
| 175 | +│ └── worker/ # Rust/Axum service, port 8082 |
| 176 | +│ ├── src/main.rs |
| 177 | +│ └── Dockerfile |
| 178 | +├── dashboard/ # SvelteKit SPA |
| 179 | +│ ├── src/ |
| 180 | +│ │ ├── lib/health.ts # Polling logic |
| 181 | +│ │ └── routes/+page.svelte # Dashboard UI |
| 182 | +│ ├── nginx.conf # Serves static build + proxies to backends |
| 183 | +│ └── Dockerfile |
| 184 | +└── helm/ |
| 185 | + └── reference-app/ # Helm chart for K8s deployment |
| 186 | + ├── Chart.yaml |
| 187 | + ├── values.yaml |
| 188 | + └── templates/ |
| 189 | +``` |
| 190 | + |
| 191 | +## API reference |
| 192 | + |
| 193 | +### API Gateway (`localhost:8080`) |
| 194 | + |
| 195 | +| Method | Path | Description | |
| 196 | +|--------|------|-------------| |
| 197 | +| GET | `/` | Service index with endpoint list | |
| 198 | +| GET | `/version` | Build metadata (version, git SHA) | |
| 199 | +| GET | `/health/live` | Liveness probe — always 200 | |
| 200 | +| GET | `/health/ready` | Readiness — checks downstream reachability | |
| 201 | +| GET | `/api/status` | Aggregated status (used by dashboard) | |
| 202 | + |
| 203 | +### Catalog (`localhost:8081`) |
| 204 | + |
| 205 | +| Method | Path | Description | |
| 206 | +|--------|------|-------------| |
| 207 | +| GET | `/` | Service index | |
| 208 | +| GET | `/version` | Build metadata | |
| 209 | +| GET | `/health/live` | Liveness probe | |
| 210 | +| GET | `/health/ready` | Readiness — checks PostgreSQL | |
| 211 | +| GET | `/items` | List all catalog items | |
| 212 | +| GET | `/items/:id` | Get single item by ID | |
| 213 | + |
| 214 | +### Worker (`localhost:8082`) |
| 215 | + |
| 216 | +| Method | Path | Description | |
| 217 | +|--------|------|-------------| |
| 218 | +| GET | `/` | Service index | |
| 219 | +| GET | `/version` | Build metadata | |
| 220 | +| GET | `/health/live` | Liveness probe | |
| 221 | +| GET | `/health/ready` | Readiness — checks PostgreSQL | |
| 222 | +| POST | `/events` | Create an event `{source, event_type, payload}` | |
| 223 | +| GET | `/events/recent` | Last 50 events | |
| 224 | + |
| 225 | +## Database schema |
| 226 | + |
| 227 | +```sql |
| 228 | +-- Catalog reads from this table |
| 229 | +CREATE TABLE items ( |
| 230 | + id SERIAL PRIMARY KEY, |
| 231 | + name VARCHAR(255) NOT NULL, |
| 232 | + description TEXT, |
| 233 | + status VARCHAR(50) NOT NULL DEFAULT 'active', |
| 234 | + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), |
| 235 | + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() |
| 236 | +); |
| 237 | + |
| 238 | +-- Worker writes to this table; heartbeat every 60s |
| 239 | +CREATE TABLE events ( |
| 240 | + id SERIAL PRIMARY KEY, |
| 241 | + source VARCHAR(100) NOT NULL, |
| 242 | + event_type VARCHAR(100) NOT NULL, |
| 243 | + payload JSONB NOT NULL DEFAULT '{}', |
| 244 | + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() |
| 245 | +); |
| 246 | +``` |
| 247 | + |
| 248 | +Connection string (Docker Compose): `postgres://refapp:refapp-lab-password@localhost:5432/refapp` |
0 commit comments