You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
📌 Lecture 8 — SRE & Monitoring: Reliability Is an Engineering Discipline
📍 Slide 1 – 💥 The Outage You Didn't Know About
🗓️ A typical Tuesday — your QuickNotes deploy goes out at 14:03. CI is green. Lab 7 deploy succeeded. You go to lunch
🚨 At 14:11, error rate jumps from 0.1% to 4.2%. p95 latency triples. Saturation creeps up
🪦 No alert fires. No one looks at a dashboard. Customers slowly notice ("the app is slow today…")
🕓 By 16:30 — when someone finally sees a graph — you've degraded service for 2.5 hours
🎓 Lesson:Code tells you what your service should do. Monitoring tells you what it actually did
🤔 Think: Lab 3 caught the bug at PR time. Lab 8 catches the bug after the deploy. Both matter. Today is the second one.
📍 Slide 2 – 🎯 Learning Outcomes
#
🎓 Outcome
1
✅ Distinguish SRE from DevOps; explain reliability as a feature
2
✅ Cite the Four Golden Signals: latency, traffic, errors, saturation
3
✅ Define SLI, SLO, SLA, error budget
4
✅ Compare metrics, logs, traces (the three pillars of observability)
5
✅ Read PromQL, set up Prometheus + Grafana for QuickNotes
6
✅ Write an alert that doesn't wake you at 3 a.m. for no reason
📍 Slide 3 – 🗺️ Lecture Overview
graph LR
A["📐 Reliability as a feature"] --> B["🎯 SLI / SLO / SLA"]
B --> C["🌟 Golden Signals"]
C --> D["📊 Prometheus"]
D --> E["📈 Grafana"]
E --> F["🚨 Alerting hygiene"]
F --> G["💥 Real Incidents"]
🪪 Error budget = freedom to take risks. Used wisely, it lets you ship at speed
💸 SLA breach often costs money. SLO breach should cost feature freeze
📍 Slide 7 – 🌟 The Four Golden Signals
From the Google SRE book, Chapter 6 — for every user-facing service, measure these:
#
Signal
What it tells you
QuickNotes example
1
⏱️ Latency
Time per request (separate success vs failure!)
p50 / p95 / p99 of GET /notes
2
🚦 Traffic
Demand on the system
Requests per second
3
🐛 Errors
Rate of failed requests
Rate of 5xx + 4xx (depends on intent)
4
📈 Saturation
How "full" the service is
CPU %, memory %, queue depth
🔎 Get these four, and you've caught 80% of production problems
🚦 The "RED" method (Tom Wilkie, Weaveworks): Rate, Errors, Duration — same idea, shortened
📍 Slide 8 – 🔍 The Three Pillars of Observability
graph TB
O["👁️ Observability"]
O --> M["📊 Metrics<br/>cheap, aggregated"]
O --> L["📜 Logs<br/>rich, expensive"]
O --> T["🧵 Traces<br/>per-request paths"]
Loading
Pillar
Best for
Tools
📊 Metrics
"How is the system now?"
Prometheus, OpenMetrics
📜 Logs
"What happened to this request?"
journald, Loki, ELK
🧵 Traces
"Where did time go across services?"
OpenTelemetry, Jaeger, Tempo
🎯 DevOps-Intro covers metrics deeply (this lecture). Logs were Lecture 4. Traces are SRE-Intro Lab 8
📍 Slide 9 – 📊 Prometheus: The Pull Model
graph LR
P["📊 Prometheus Server"] -- "HTTP GET /metrics every 15s" --> S1["🟢 QuickNotes (:8080)"]
P -- "scrape" --> S2["🟢 node-exporter"]
P --> TSDB["⏳ Time-series DB"]
G["📈 Grafana"] -- "PromQL" --> P
Loading
🆕 Prometheus (CNCF, graduated 2018) pulls metrics from targets — opposite of push-based StatsD/graphite
🆔 Each target exposes /metrics in text format (your QuickNotes already does — see handlers.go)
⏳ Stored as time series — metric_name{label1="v",label2="v"} value timestamp
🆓 Open source, single binary, no clustering required for small setups
📍 Slide 10 – 📐 PromQL: The Three Queries You Need
# 1) Instant rate of requests over the last minute
rate(quicknotes_http_requests_total[1m])
# 2) p95 latency of /notes
histogram_quantile(0.95,
sum by (le) (rate(quicknotes_http_request_duration_seconds_bucket{route="/notes"}[5m]))
)
# 3) Error ratio (5xx + 4xx) over 5 minutes
sum(rate(quicknotes_http_responses_by_code_total{code=~"5..|4.."}[5m]))
/
sum(rate(quicknotes_http_requests_total[5m]))
🔧 Operators:rate(), sum by(), histogram_quantile, </> for alert thresholds
🪤 rate() works on counters only; for gauges use avg_over_time() / delta()
🎁 Lab 8 Bonus task wires Checkly free-tier against your deployed QuickNotes
📍 Slide 14 – 🩺 The On-Call Mindset
graph LR
A["🚨 Alert fires"] --> B["📜 Runbook"]
B --> C{Mitigate?}
C -- "yes" --> D["🪡 Fix the symptom now"]
D --> E["📝 Postmortem"]
C -- "no" --> F["🆘 Escalate"]
E --> G["🛠️ Action items<br/>(prevent next time)"]
Loading
🎯 First mitigate, then fix. A 5-minute hack that stops the bleeding beats a 5-hour root-cause fix
📝 Every page → a postmortem (Lecture 1 covered the blameless format). The action items are the real value
🤝 Healthy rotations: ≤ 1 page per shift average, week-on/week-off, follow-the-sun across timezones
📍 Slide 15 – 📜 Real Story: The Slack 2022 Outage
🗓️ February 22, 2022 (yes, "2/22/22") — Slack offline for ~5 hours
🤖 Triggered by an automated config push that broke a database. Cascading failures took out web, messaging, file uploads
🩺 The status page itself struggled — hosted on the same infrastructure
📊 Task 1 (6 pts): Run Prometheus + Grafana via Compose against your QuickNotes container. Provision a "Golden Signals" dashboard (rate, error %, p95 latency, memory)
🚨 Task 2 (4 pts): Write one good alert: error rate > 5% for 5 minutes. Trigger it deliberately by setting FAIL_RATE=1.0. Verify the alert fires, document the runbook step
🌍 Bonus (2 pts): Spin up a Checkly free-tier synthetic check against your QuickNotes (deployed via Lab 7 or running locally with ngrok). Compare external view vs internal metrics
📜 Deliverable: submissions/lab8.md — dashboard screenshots, alert evidence, written analysis
📍 Slide 18 – 🧠 Key Takeaways
📐 Reliability is a feature — budget for it, like any other feature
🌟 The Four Golden Signals (Latency, Traffic, Errors, Saturation) catch 80% of incidents
🎯 SLI → SLO → error budget is the framework that makes "reliable enough" measurable
📊 Metrics + Logs + Traces are complementary, not redundant
🚨 Fewer, sharper alerts beat many noisy ones — alert on user-visible symptoms
📝 Every page → a runbook → a postmortem — closing the loop is how systems get better
📍 Slide 19 – 🚀 What's Next + 📚 Resources
📍 Next lecture: DevSecOps — shift security left; scan deps and images
🧪 Lab 8: Prometheus + Grafana for QuickNotes, one good alert, Bonus: Checkly synthetic
graph LR
P["🔧 Week 7<br/>Ansible CM"] --> Y["📍 You Are Here<br/>SRE & Monitoring"]
Y --> N["🛡️ Week 9<br/>DevSecOps"]
N --> M["☁️ Week 10<br/>Cloud"]
Loading
🎯 Remember: A service without monitoring is a service running in the dark. With the Four Golden Signals dashboard up, every incident response starts with "what does the data say?" — not "who knows the most about this code?"