Skip to content

Commit bcbc61e

Browse files
cliffablecliffable
committed
Add monitoring and backups Field Notes and feature Field Notes on homepage
1 parent 2061375 commit bcbc61e

9 files changed

Lines changed: 972 additions & 11 deletions

File tree

‎assets/images/field-notes/from-silent-failures-to-automated-alerting/cloudwatch-alarm-email-alert.png‎

225 KB
Loading

‎assets/images/field-notes/from-silent-failures-to-automated-alerting/csgl-lightsail-production-architecture.png‎

814 KB
Loading

‎assets/images/field-notes/rethinking-backups-after-a-failed-restore/csgl-lightsail-architecture.png‎

691 KB
Loading

‎assets/images/field-notes/rethinking-backups-after-a-failed-restore/csgl-lightsail-production-architecture.png‎

814 KB
Loading

‎field-notes/from-silent-failures-to-automated-alerting.html‎

Lines changed: 344 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,344 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
4+
<head>
5+
<!--
6+
FIELD NOTE TEMPLATE
7+
Duplicate this file for new Field Notes articles.
8+
-->
9+
10+
<meta charset="UTF-8">
11+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
12+
13+
<title>From Silent Failures to Automated Alerting | Cliffable</title>
14+
15+
<meta name="description"
16+
content="How intermittent WordPress outages after a Lightsail migration led to swap configuration, health checks, CloudWatch alarms and SNS email alerts.">
17+
18+
<meta name="robots" content="index, follow">
19+
20+
<link rel="canonical"
21+
href="https://cliffable.com/field-notes/from-silent-failures-to-automated-alerting.html">
22+
23+
<!-- Google Fonts -->
24+
<link rel="preconnect" href="https://fonts.googleapis.com">
25+
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
26+
<link href="https://fonts.googleapis.com/css2?family=Montserrat:wght@400;600&display=swap" rel="stylesheet">
27+
28+
<link rel="stylesheet" href="../styles.css">
29+
30+
<!-- Favicons -->
31+
<link rel="icon" type="image/x-icon" href="../favicon/favicon.ico">
32+
<link rel="icon" type="image/png" sizes="32x32" href="../favicon/favicon-32x32.png">
33+
<link rel="icon" type="image/png" sizes="16x16" href="../favicon/favicon-16x16.png">
34+
<link rel="apple-touch-icon" sizes="180x180" href="../favicon/apple-touch-icon.png">
35+
36+
<!-- Google Analytics -->
37+
<script async src="https://www.googletagmanager.com/gtag/js?id=G-L41X6NGBMK"></script>
38+
39+
<script>
40+
window.dataLayer = window.dataLayer || [];
41+
42+
function gtag() {
43+
dataLayer.push(arguments);
44+
}
45+
46+
gtag('js', new Date());
47+
48+
gtag('config', 'G-L41X6NGBMK');
49+
</script>
50+
51+
</head>
52+
53+
<body>
54+
55+
<header>
56+
<div class="container header-inner">
57+
58+
<div class="logo">
59+
<a href="/">
60+
<img src="/assets/logos/cliffable-logo-white-v2.svg"
61+
alt="Cliffable logo"
62+
height="60">
63+
</a>
64+
</div>
65+
66+
<nav class="main-nav">
67+
<a href="/projects/">Projects</a>
68+
<a href="/field-notes/">Field Notes</a>
69+
<a href="/about.html">About</a>
70+
<a href="/contact.html">Contact</a>
71+
</nav>
72+
73+
</div>
74+
</header>
75+
76+
<main>
77+
78+
<!-- HERO -->
79+
80+
<section class="hero field-note-article-hero">
81+
<div class="container field-note-container">
82+
83+
<p class="field-note-article-meta">
84+
Operations • Monitoring • AWS
85+
</p>
86+
87+
<p class="field-note-date">
88+
Published May 2026
89+
</p>
90+
91+
<h1>From Silent Failures to Automated Alerting</h1>
92+
93+
<p class="field-note-intro">
94+
After migrating a production WordPress site to the official AWS Lightsail WordPress blueprint, the
95+
site went down twice without any automatic warning. That changed how I thought about monitoring,
96+
stability and operational visibility.
97+
</p>
98+
99+
</div>
100+
</section>
101+
102+
<!-- FEATURED IMAGE -->
103+
104+
<section class="field-note-featured-image">
105+
<div class="container field-note-container">
106+
107+
<a href="../assets/images/field-notes/from-silent-failures-to-automated-alerting/csgl-lightsail-production-architecture.png"
108+
class="lightbox-trigger">
109+
110+
<img src="../assets/images/field-notes/from-silent-failures-to-automated-alerting/csgl-lightsail-production-architecture.png"
111+
alt="AWS Lightsail WordPress production architecture with monitoring and alerting">
112+
113+
</a>
114+
115+
<p class="field-note-image-caption">
116+
Production Lightsail WordPress architecture with Route 53 health checks, CloudWatch alarms, SNS
117+
notifications and snapshot-based recovery.
118+
</p>
119+
120+
</div>
121+
</section>
122+
123+
<!-- ARTICLE CONTENT -->
124+
125+
<section class="field-note-content-section">
126+
<div class="container field-note-container">
127+
128+
<h2>The problem with silent failures</h2>
129+
130+
<p>
131+
After migrating the site from the deprecated Bitnami Lightsail WordPress blueprint to the official
132+
AWS Lightsail WordPress blueprint, the production site became intermittently unstable.
133+
</p>
134+
135+
<p>
136+
On two separate occasions, the site went down silently. I only noticed because I happened to check
137+
the website manually.
138+
</p>
139+
140+
<p>
141+
That was the real issue. The outage itself mattered, but the bigger operational problem was that the
142+
system had no reliable way to tell me something had failed.
143+
</p>
144+
145+
<h2>Investigating the instability</h2>
146+
147+
<p>
148+
The site was running on a 1 GB Lightsail instance with no swap configured. Investigation suggested
149+
that the failures were most likely caused by temporary memory pressure during WordPress, Apache or
150+
background maintenance activity.
151+
</p>
152+
153+
<p>
154+
A small production site can appear stable most of the time, but short bursts of activity can still
155+
exhaust available memory. Without swap, the instance had less room to absorb those temporary spikes.
156+
</p>
157+
158+
<p>
159+
To improve stability under memory pressure, I configured a persistent 1 GB swap file on the server.
160+
</p>
161+
162+
<h2>Adding external monitoring and alerts</h2>
163+
164+
<p>
165+
The next priority was visibility. I wanted the system to alert me automatically if the site went down
166+
again, rather than relying on chance discovery.
167+
</p>
168+
169+
<p>
170+
I implemented Route 53 HTTPS health checks for external uptime monitoring, then connected the health
171+
check status to CloudWatch alarms and SNS email notifications.
172+
</p>
173+
174+
<p>
175+
I also added Lightsail alarms for key instance-level signals, including CPU utilisation, burst
176+
capacity and status check failures.
177+
</p>
178+
179+
<ul>
180+
<li>Route 53 HTTPS health checks monitor whether the public website is reachable</li>
181+
<li>CloudWatch alarms detect unhealthy states and metric thresholds</li>
182+
<li>SNS sends email notifications when alarms move into the ALARM state</li>
183+
<li>Lightsail alarms provide instance-level visibility into resource pressure and status failures</li>
184+
</ul>
185+
186+
<h2>Testing the alerting path</h2>
187+
188+
<p>
189+
I did not want to assume the alerts worked just because they were configured.
190+
</p>
191+
192+
<p>
193+
To validate the full monitoring path, I tested the setup using a real Apache outage. The site became
194+
unavailable, the health check failed, the CloudWatch alarm moved into the ALARM state, and SNS sent
195+
an email notification.
196+
</p>
197+
198+
<!-- OPTIONAL INLINE IMAGE -->
199+
200+
<div class="field-note-inline-image">
201+
202+
<a href="../assets/images/field-notes/from-silent-failures-to-automated-alerting/cloudwatch-alarm-email-alert.png"
203+
class="lightbox-trigger">
204+
205+
<img src="../assets/images/field-notes/from-silent-failures-to-automated-alerting/cloudwatch-alarm-email-alert.png"
206+
alt="CloudWatch alarm email notification showing a website down alert">
207+
208+
</a>
209+
210+
<p class="field-note-image-caption">
211+
CloudWatch alarm email generated through SNS after validating the website-down alerting path.
212+
</p>
213+
214+
</div>
215+
216+
<p>
217+
That test was important because it confirmed the complete detection chain, not just the individual
218+
AWS resources.
219+
</p>
220+
221+
<h2>What changed most</h2>
222+
223+
<p>
224+
Before this work, production failures depended on me noticing them manually.
225+
</p>
226+
227+
<p>
228+
After adding health checks, alarms and email notifications, the system became capable of reporting
229+
failures automatically.
230+
</p>
231+
232+
<p>
233+
The main improvement was not just technical stability. It was operational awareness.
234+
</p>
235+
236+
<h2>Conclusion</h2>
237+
238+
<p>
239+
This was a useful reminder that keeping a production site online is not only about the server
240+
running successfully most of the time. It is also about knowing quickly when something has gone
241+
wrong.
242+
</p>
243+
244+
<p>
245+
Adding swap improved resilience under memory pressure, while Route 53 health checks, CloudWatch
246+
alarms, Lightsail alarms and SNS notifications improved visibility. The site moved from silent
247+
failure to monitored operation.
248+
</p>
249+
250+
<!-- RELATED PROJECT -->
251+
252+
<h2>Related project</h2>
253+
254+
<p class="field-note-callout">
255+
This work was implemented as part of the
256+
<a href="../projects/csgl-lightsail-wordpress.html">
257+
WordPress on AWS Lightsail
258+
</a>
259+
project.
260+
</p>
261+
262+
</div>
263+
</section>
264+
265+
</main>
266+
267+
<!-- FOOTER -->
268+
269+
<footer>
270+
<div class="container footer-content">
271+
272+
<div class="footer-contact">
273+
274+
<p class="footer-heading">Contact</p>
275+
276+
<div class="footer-links">
277+
278+
<a href="mailto:mailcliffsmith@gmail.com">
279+
Email
280+
</a>
281+
282+
<a href="https://www.linkedin.com/in/cliff-smith-london/"
283+
target="_blank"
284+
rel="noopener noreferrer">
285+
LinkedIn
286+
</a>
287+
288+
<a href="https://github.com/cliffable"
289+
target="_blank"
290+
rel="noopener noreferrer">
291+
GitHub
292+
</a>
293+
294+
</div>
295+
</div>
296+
297+
<div class="footer-meta">
298+
<p>© 2026 Cliff Smith</p>
299+
</div>
300+
301+
</div>
302+
</footer>
303+
304+
<!-- LIGHTBOX -->
305+
306+
<div id="lightbox" class="lightbox">
307+
<img id="lightbox-img" src="" alt="">
308+
</div>
309+
310+
<script>
311+
312+
const lightboxTriggers = document.querySelectorAll('.lightbox-trigger');
313+
314+
const lightbox = document.getElementById('lightbox');
315+
316+
const lightboxImg = document.getElementById('lightbox-img');
317+
318+
lightboxTriggers.forEach(trigger => {
319+
320+
trigger.addEventListener('click', event => {
321+
322+
event.preventDefault();
323+
324+
lightbox.style.display = 'flex';
325+
326+
lightboxImg.src = trigger.href;
327+
328+
});
329+
330+
});
331+
332+
lightbox.addEventListener('click', () => {
333+
334+
lightbox.style.display = 'none';
335+
336+
lightboxImg.src = '';
337+
338+
});
339+
340+
</script>
341+
342+
</body>
343+
344+
</html>

0 commit comments

Comments
 (0)