Skip to content

Commit dad8ac7

Browse files
committed
chore: edits
1 parent 9adbc28 commit dad8ac7

1 file changed

Lines changed: 5 additions & 7 deletions

File tree

  • content/posts/2026/closing-the-alert-is-not-enough

content/posts/2026/closing-the-alert-is-not-enough/index.md

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ If the checkout service starts timing out, you can often find the obvious broken
2222

2323
But clearing the alert only proves that the immediate symptom went away. It does not prove that the team understands the failure.
2424

25-
A complete investigation answers three questions: what failed, what allowed it to fail, and what old constraint made the broken design seem reasonable. Skipping any of them leaves you guessing.
25+
A complete investigation answers three questions: what failed, what allowed it to fail, and **what old constraint made the broken design seem reasonable**.
2626

2727
## Phase 1: What failed
2828

@@ -60,7 +60,7 @@ In the checkout example, the immediate problem might be a full table scan. The q
6060

6161
So you keep digging.
6262

63-
The query slowed down because an automated migration cleanup tool dropped an index it flagged as unused. The tool flagged it as unused because it only analyzed recent production traffic. It missed the seasonal promotion path, which only runs during large campaigns and uses a different filter pattern.
63+
The query slowed down because an engineer deliberately dropped the index to save database storage and improve write latency. A database observability tool had flagged the index as unused over a 30-day window, prompting a cleanup PR. The reviewer saw the telemetry recommendation and approved it. Both of them missed the seasonal promotion path, which only runs during large yearly campaigns and relies entirely on that specific index.
6464

6565
The failed query matters, but the query is not the whole story. The dangerous condition was that the index looked unused to automation but was still required by a low-frequency business process. The migration review process did not catch that distinction. The available index-usage data did not make it visible. The tests did not include the data shape that made the query expensive.
6666

@@ -82,7 +82,7 @@ Once you find the broken code, the temptation is to clean it up. Delete the weir
8282

8383
Sometimes that is exactly right. Sometimes the old code really is just wrong.
8484

85-
But code rarely enters a codebase as random nonsense. It usually solved a real problem under constraints that existed at the time. Those constraints may be gone now. They may still exist. You need to know which.
85+
But code rarely enters a codebase as random nonsense. *Well, maybe it does a bit more, now...* But usually all code solved a real problem under constraints that existed at the time. Those constraints may be gone now or this section is code is being executed for purposes it was never intended for originally.
8686

8787
Before you change the strange part, dig into the history. Run [`git blame`](https://git-scm.com/docs/git-blame), read the old pull request, search for the related ticket, and check whether there was an architecture decision record, or [ADR](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions). You are reconstructing the original engineer's mental model.
8888

@@ -101,7 +101,7 @@ Or maybe it is nonsense.
101101

102102
You do not know until you check.
103103

104-
The same applies to the checkout incident. Why did that index exist in the first place? Why did the seasonal promotion path use a different filter pattern? Why did the migration cleanup tool trust recent traffic as a complete signal? Why did nobody document that the index was tied to campaign traffic?
104+
The same applies to the checkout incident. Why did that index exist in the first place? Why did the seasonal promotion path use a different filter pattern? Why did the engineering team trust a 30-day metric as a complete signal for safe deletion? Why did nobody document that the index was tied strictly to Black Friday traffic?
105105

106106
Those questions can be uncomfortable because they reveal old tradeoffs. Maybe the team moved fast because the promotion system had to launch before Black Friday. Maybe the seasonal path was supposed to be temporary. Maybe the database was small enough at the time that nobody cared about the query plan. Maybe the engineer who knew all of this left two years ago, and the only remaining documentation is a comment in a migration file that says "needed for campaign traffic."
107107

@@ -115,8 +115,6 @@ Sometimes the conclusion is: this made sense then, but it does not anymore.
115115

116116
That is a good outcome. Now you can remove it deliberately, document why the old constraint no longer applies, and leave the next person a better trail than the one you found.
117117

118-
Not every alert requires a week of archaeology, but stopping at the failing line leaves the system ready to surprise you again.
119-
120-
The depth of the investigation should match the risk, but the habit should be the same: do not confuse the line that failed with the reason the system failed.
118+
Not every alert requires a week of archaeology, but stopping at the immediate fix guarantees the system will surprise you again.
121119

122120
Fixing the code restores the service. Understanding the context fixes the system.

0 commit comments

Comments
 (0)