|
| 1 | +# Investigating Alerting Behavior |
| 2 | + |
| 3 | +Use this read-only flow when a user asks which alert settings actually apply to |
| 4 | +a deployed check, especially when check-local, group, global, and alert-channel |
| 5 | +settings may interact. |
| 6 | + |
| 7 | +Table output is insufficient for this scenario. Use JSON or raw API evidence |
| 8 | +whenever available because human tables can omit `alertSettings`, |
| 9 | +`useGlobalAlertSettings`, `alertChannelSubscriptions`, `groupId`, `muted`, |
| 10 | +`activated`, `retryStrategy`, and `doubleCheck`. |
| 11 | + |
| 12 | +Only use read-only commands. Do not deploy, update, delete, trigger checks, |
| 13 | +start test runs, run RCA, mutate incidents, or change alert channels. When using |
| 14 | +`checkly api`, only use `GET` requests; do not pass write methods, `-F` fields, |
| 15 | +or request bodies. |
| 16 | + |
| 17 | +Keep the investigation bounded. Use the commands and endpoints listed in this |
| 18 | +reference, plus exact endpoints returned by those commands if they are needed to |
| 19 | +follow a concrete ID. Do not guess multiple possible account/global alerting |
| 20 | +endpoints. If a listed command or endpoint does not expose a field, report that |
| 21 | +the field was unavailable in the inspected output instead of probing variants. |
| 22 | + |
| 23 | +## Recommended Read-Only Command Sequence |
| 24 | + |
| 25 | +Start by identifying one concrete check. If the user gave a name instead of an |
| 26 | +ID, search and inspect the JSON list output before choosing: |
| 27 | + |
| 28 | +```bash |
| 29 | +checkly checks list --output json --limit 100 |
| 30 | +checkly checks list --output json --limit 100 --search "<check-name>" |
| 31 | +``` |
| 32 | + |
| 33 | +Then inspect the selected check with structured output: |
| 34 | + |
| 35 | +```bash |
| 36 | +checkly checks get <check-id> --output json |
| 37 | +checkly api /v1/checks/<check-id> |
| 38 | +``` |
| 39 | + |
| 40 | +Use the CLI detail output for context only after you have captured JSON/API |
| 41 | +evidence: |
| 42 | + |
| 43 | +```bash |
| 44 | +checkly checks get <check-id> |
| 45 | +``` |
| 46 | + |
| 47 | +Inspect alert channels as structured evidence. The list helps identify channel |
| 48 | +IDs and subscription scopes. Only call `get` for channel IDs referenced by the |
| 49 | +selected check or matching group subscriptions, or for a user-named channel you |
| 50 | +must verify. Do not fetch every channel when there are no relevant |
| 51 | +subscriptions: |
| 52 | + |
| 53 | +```bash |
| 54 | +checkly alert-channels list --output json --limit 100 |
| 55 | +checkly alert-channels get <alert-channel-id> --output json |
| 56 | +``` |
| 57 | + |
| 58 | +If the check has a `groupId`, fetch groups once and locate the matching group. |
| 59 | +Use the raw API because group override fields may not have a dedicated CLI |
| 60 | +command: |
| 61 | + |
| 62 | +```bash |
| 63 | +checkly api /v1/check-groups |
| 64 | +``` |
| 65 | + |
| 66 | +If the selected check has no `groupId`, or if |
| 67 | +`checks list --output json --limit 100` returns no checks with `groupId`, stop |
| 68 | +group override investigation. Say that group override analysis is unavailable |
| 69 | +from the current evidence and continue with check, global-selection, and |
| 70 | +alert-channel evidence. Do not invent group behavior and do not keep looking for |
| 71 | +alternative group endpoints. |
| 72 | + |
| 73 | +This reference does not define a dedicated account/global alert-settings |
| 74 | +endpoint. When `useGlobalAlertSettings: true` is the only account/global signal, |
| 75 | +say that global alert settings are selected but their policy details were not |
| 76 | +available in the inspected output. Do not probe guessed endpoints such as |
| 77 | +account-settings, account/settings, check-alerts, or alert-notifications just to |
| 78 | +fill that gap. |
| 79 | + |
| 80 | +## Fields To Inspect |
| 81 | + |
| 82 | +For the selected check, capture and explain: |
| 83 | + |
| 84 | +- `activated` - whether the check is scheduled to run. |
| 85 | +- `muted` - whether check-level alert notifications are suppressed. |
| 86 | +- `groupId` - whether group settings may affect the check. |
| 87 | +- `alertSettings` - check-local escalation policy evidence. |
| 88 | +- `useGlobalAlertSettings` - whether the check selects account/global alert |
| 89 | + settings instead of a local policy. |
| 90 | +- `alertChannelSubscriptions` - check-local or group-scoped channel bindings, |
| 91 | + including each subscription's `activated`, `checkId`, `groupId`, and channel |
| 92 | + ID. |
| 93 | +- `retryStrategy` and `doubleCheck` - retry or double-check behavior that can |
| 94 | + delay or change when a failure becomes alert-worthy. |
| 95 | + |
| 96 | +For a matching group, capture and explain: |
| 97 | + |
| 98 | +- `activated` and `muted` - whether the group can stop runs or suppress alerts. |
| 99 | +- `alertSettings` and `useGlobalAlertSettings` - whether the group forces a |
| 100 | + custom or global escalation policy for grouped checks. |
| 101 | +- `alertChannelSubscriptions`, or alert-channel subscriptions whose `groupId` |
| 102 | + matches the group - which group-scoped channels are active. |
| 103 | +- `retryStrategy` and `doubleCheck` - group-level retry behavior when present. |
| 104 | + |
| 105 | +For alert channels, capture and explain: |
| 106 | + |
| 107 | +- channel `id`, `type`, name/config label, and whether the channel itself is |
| 108 | + active if the API exposes that field. |
| 109 | +- subscription scope: check-local (`checkId` matches the check and `groupId` is |
| 110 | + empty), group-scoped (`groupId` matches the check's group), or unrelated. |
| 111 | +- subscription `activated` status. Inactive subscriptions are evidence of |
| 112 | + configuration, not active alert delivery. |
| 113 | + |
| 114 | +## Effective Alerting Decision Tree |
| 115 | + |
| 116 | +Apply this tree from confirmed evidence only: |
| 117 | + |
| 118 | +1. Check run and notification gates: |
| 119 | + - If the check has `activated: false`, report that it is inactive and should |
| 120 | + not produce scheduled alerts unless other output proves otherwise. |
| 121 | + - If the check has `muted: true`, report that check-level notifications are |
| 122 | + suppressed. |
| 123 | + - If a fetched group for this check has `activated: false`, report that the |
| 124 | + group stops grouped checks from running when the API output confirms it. |
| 125 | + - If a fetched group has `muted: true`, report that group-level muting |
| 126 | + suppresses alerts for grouped checks. |
| 127 | +2. Group override status: |
| 128 | + - If the check has no `groupId`, group overrides do not apply to this check. |
| 129 | + - If no grouped checks are present in the inspected list output, group |
| 130 | + override analysis is unavailable from current evidence. |
| 131 | + - If the check has a `groupId`, inspect the matching group before deciding |
| 132 | + effective escalation or group channel behavior. |
| 133 | + - If group `useGlobalAlertSettings` is `false` and group `alertSettings` is |
| 134 | + present, report the group escalation policy as the effective policy for |
| 135 | + the grouped check. |
| 136 | + - If group `useGlobalAlertSettings` is `true`, report that the group selects |
| 137 | + account/global escalation. Do not describe the global policy unless another |
| 138 | + output shows its values. |
| 139 | + - If group `useGlobalAlertSettings` is missing, `null`, or `undefined`, and |
| 140 | + no group `alertSettings` is present, do not claim a group escalation |
| 141 | + override. Continue to check-local evidence. |
| 142 | + - If group-scoped alert channel subscriptions are present and active, report |
| 143 | + them as group-scoped channels. If group channels exist but group policy |
| 144 | + fields are unavailable, say whether channel applicability is unavailable |
| 145 | + rather than assuming they apply. |
| 146 | +3. Check-local escalation: |
| 147 | + - If check `useGlobalAlertSettings` is `false` and check `alertSettings` is |
| 148 | + present, report the check-local escalation policy as effective unless a |
| 149 | + confirmed group override above supersedes it. |
| 150 | + - If check `useGlobalAlertSettings` is `true`, report that the check selects |
| 151 | + account/global escalation unless a confirmed group override above supersedes |
| 152 | + it. |
| 153 | + - If check `alertSettings` and `useGlobalAlertSettings` are absent or |
| 154 | + contradictory, report the escalation source as unavailable instead of |
| 155 | + inferring defaults. |
| 156 | +4. Account/global settings: |
| 157 | + - Treat account/global settings as known only when CLI or API output exposes |
| 158 | + their values. |
| 159 | + - Do not try multiple guessed account/global alerting endpoints. Use a |
| 160 | + documented endpoint only when this reference, CLI output, or the API |
| 161 | + reference gives the exact path. If that one documented request fails or does |
| 162 | + not include the needed fields, report the values as unavailable. |
| 163 | + - If output only shows `useGlobalAlertSettings: true`, say "global alert |
| 164 | + settings are selected, but their policy details were not available in the |
| 165 | + inspected CLI/API output." |
| 166 | +5. Retry and double-check behavior: |
| 167 | + - If `retryStrategy` is present, explain how it affects when alerts can fire |
| 168 | + based on the returned fields. |
| 169 | + - If `doubleCheck` is present and `retryStrategy` is absent, explain the |
| 170 | + double-check behavior as returned. |
| 171 | + - If neither field is available, say retry or double-check behavior was not |
| 172 | + available in the inspected output. |
| 173 | + |
| 174 | +## Response Template |
| 175 | + |
| 176 | +Use this structure when reporting back: |
| 177 | + |
| 178 | +```markdown |
| 179 | +## Alerting evidence for <check-name> (<check-id>) |
| 180 | + |
| 181 | +### Confirmed evidence |
| 182 | +- Check state: activated=<value>, muted=<value>, groupId=<value> |
| 183 | +- Check escalation fields: useGlobalAlertSettings=<value>, alertSettings=<summary or null> |
| 184 | +- Retry fields: retryStrategy=<summary or unavailable>, doubleCheck=<value or unavailable> |
| 185 | +- Sources: <commands used> |
| 186 | + |
| 187 | +### Effective escalation policy |
| 188 | +<State the effective escalation policy source: group custom policy, group-selected |
| 189 | +global policy, check-local policy, check-selected global policy, or unavailable. |
| 190 | +Quote only values visible in JSON/API output.> |
| 191 | + |
| 192 | +### Alert channels |
| 193 | +- Check-local channels: <active channel IDs/types/names, inactive subscriptions, or none> |
| 194 | +- Group-scoped channels: <active channel IDs/types/names, inactive subscriptions, unavailable, or not applicable> |
| 195 | +- Unrelated channels ignored: <short summary if useful> |
| 196 | + |
| 197 | +### Group override status |
| 198 | +<No groupId / no grouped checks found / matching group inspected with fields / |
| 199 | +matching group unavailable. Explain whether group settings or channels override |
| 200 | +or affect the check based only on confirmed fields.> |
| 201 | + |
| 202 | +### Unavailable evidence |
| 203 | +- Account/global alert policy details: <known values or unavailable> |
| 204 | +- Missing fields/endpoints: <fields not present or API data not returned> |
| 205 | + |
| 206 | +### Recommended next read-only checks |
| 207 | +- <Only include GET/list commands that would close evidence gaps, such as |
| 208 | + `checkly api /v1/checks/<check-id>` or |
| 209 | + `checkly alert-channels get <alert-channel-id> --output json` for a channel |
| 210 | + ID referenced by a relevant subscription. Do not recommend guessed |
| 211 | + account/global alerting endpoints.> |
| 212 | +``` |
0 commit comments