Skip to content

incident-response.md

Latest commit

 

History

History
192 lines (149 loc) · 8.17 KB

incident-response.md

File metadata and controls

192 lines (149 loc) · 8.17 KB
Raw
title Incident response
description Symptom-to-action mapping with cross-links to the R1-R7 runbooks.
tags
incident-response
runbooks

Incident response

3am-tuned symptom → action mapping. Use this page as the first read after the alert fires; each row points at a runbook (R1–R7) for the full playbook.

If you are reading this BEFORE an incident: also look at troubleshooting.md for symptom-keyed diagnostics and at the runbook index for the seven full disaster runbooks.

First five minutes

Whatever fired, first run:

pg_hardstorage doctor
pg_hardstorage status

doctor returns one finding per deployment with a suggested fix. status returns RPO and the next scheduled run for each deployment. Together they answer "what's broken and how recent is the data we'd restore from?" — the two questions that gate every other decision.

If doctor exits 10 (doctor.issues_present), run the command doctor suggests for each finding, one at a time, after acknowledging each.

Symptom matrix

Alert / symptom Most likely cause First action Runbook
HSWALSilence, wal.slot_missing Replication slot dropped pg_hardstorage wal repair <deployment> R6
HSWALSilence, slot present, lag rising Network partition or PG primary stuck Check pg_replication_slots; verify libpq reachability R6
HSBackupOverdue, RPO target exceeded Schedule paused / agent down pg_hardstorage status <deployment>; rerun backup manually R3
HSKEKUnreachable, kms.unreachable KMS provider degraded or IAM revoked Check provider console; restore IAM grants R2
HSScrubFindings{kind="bit-rot"} Chunk corruption at rest Quarantine prefix; heal from replica region R4
HSScrubFindings{kind="missing"} Manifest references absent chunk repo check to confirm; restore from replica R4
HSVerifyFailures on tier="fast" Encryption or storage corruption Cross-check a different backup; if also failing, treat as repo corruption R4
HSVerifyFailures on tier="full" Backup unrestorable; PG-level damage Promote a different backup; investigate source database R3
verify.audit_chain_broken Audit-log tamper or storage corruption audit verify-chain for the bad event ID; freeze repo R4
repo region gone (S3 bucket deleted, region outage) Disaster scenario Promote replica region; rotate KEK refs R1
kms.shred accidentally executed Backups for tenant T unrecoverable Document affected backups; honour as compliance event; restore older snapshots if any predate the KEK R2
Restore halted with state file present in target Half-applied PITR Roll back to staging; rerun with --target clean R5
HSAnomalyHigh{kind="size"} Table dropped, bulk import, replication broken Compare manifest sizes via backup compare; investigate workload troubleshooting.md
Patroni split-brain detected Two primaries — refuse all backups Resolve in DCS first; resume only after lone leader R7

Decision tree

graph TD
    A[Alert fires] --> B{doctor exits 10?}
    B -->|yes| C[Apply suggested fixes]
    B -->|no| D{Which class?}
    C --> E[Re-run doctor; if clean, monitor]
    D -->|WAL/RPO| F[R6: slot/gap]
    D -->|Repo corruption| G[R4: corruption at rest]
    D -->|KMS| H[R2: KMS key destroyed]
    D -->|Region down| I[R1: repo region gone]
    D -->|Restore stuck| J[R5: half-applied PITR]
    D -->|Patroni| K[R7: split-brain]
    D -->|Cold start| L[R3: cold start from backups]
Loading

Triage commands cheat-sheet

# Health
pg_hardstorage doctor [<deployment>]
pg_hardstorage status [<deployment>]

# Inspect a specific backup
pg_hardstorage list <deployment>
pg_hardstorage show <deployment> <backup-id>

# Verify on the spot
pg_hardstorage verify <deployment> latest --repo <url>
pg_hardstorage repair scrub <repo-url>

# Audit chain integrity
pg_hardstorage audit verify-chain --repo <url>
pg_hardstorage audit search --deployment <d> --since 24h

# WAL diagnosis
pg_hardstorage wal list <d> --repo <url>
pg_hardstorage wal repair <deployment>
pg_hardstorage wal gaps <d> --repo <url>

# Capacity / cost (post-incident retrospective)
pg_hardstorage capacity report --repo <url>
pg_hardstorage cost report --repo <url>

Pre-flight refusals (refuse-don't-corrupt)

The binary refuses to mutate state in known-broken configurations. The refusals are the first defence — many incidents resolve by reading the refusal and running the suggested fix.

Refusal code Meaning Suggestion
verify.signature_mismatch Manifest's Ed25519 signature failed Verify the keyring; restore from replica
verify.scrub_mismatch Chunk's plaintext SHA-256 disagrees with manifest R4
verify.audit_chain_broken Audit chain's hash links don't match Freeze repo; investigate tamper
verify.missing_chunks Live manifest references a chunk not in CAS R4
wal.slot_missing Replication slot dropped on primary wal repair <deployment>
wal_gap_detected Recorded gap in WAL inventory Decide: accept loss + new full, or use wal repair
pg.connect_timeout libpq could not connect within 30s Check pg_hba.conf, firewall, port
kms.unreachable KMS provider unreachable Wait or fail over; do not force-mutate
patroni.split_brain Two leaders in DCS R7
repo.tombstoned Operation targets a tombstoned manifest Check rotate history; undelete if appropriate

Every refusal carries a Suggestion field with the exact remediation command. Refusals are exit code 4 (pre-flight failed) and never mutate state.

Post-incident

Once the incident is resolved:

  1. Capture forensic state. Export an audit evidence bundle that covers the incident window:

    pg_hardstorage audit export-bundle \
    --repo <url> \
    --since "2026-04-28T03:00:00Z" \
    --until "2026-04-28T09:00:00Z" \
    --include-anchors \
    --out ./incident-2026-04-28.tar.gz

    See audit evidence bundles for the format.

  2. Run a drill. Trigger an out-of-cycle recovery drill into a different region:

    pg_hardstorage recovery drill <deployment>

    The drill records actual RTO vs SLO target.

  3. Update the SLO. If the SLO target was missed, update it to reflect actual capacity rather than aspirational targets:

    pg_hardstorage slo set <deployment> --rpo 4h --rto 30m

  4. Schedule a retrospective. The audit chain plus the evidence bundle is your write-up's primary source.

Further reading