Skip to content

Commit e3fa26b

Browse files
thebiglabaskyclaudestefanjudis
authored
feat(cli): add investigate + communicate actions and confirmation protocol to AI skills (#1247)
* feat(cli): add manage action with confirmation protocol to AI skills Adds a `manage` action to `checkly skills` covering operational commands (checks list/get, incidents lifecycle, status pages) and documents the confirmation protocol for write commands. - Adds confirmation protocol note to SKILL.md (always in context) - Adds manage.md with full protocol docs and rules for agents - Adds manage-checks.md and manage-incidents.md references - Updates prepare-ai-context.ts to process manage references Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update generated skills/checkly/SKILL.md Regenerated from prepare-ai-context to include the manage action and confirmation protocol section. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add AI context regeneration workflow to CONTRIBUTING.md Documents the required steps when modifying agent skill sources: run prepare, copy generated SKILL.md, and commit both. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(cli): align AI skills with RFC — split manage into investigate + communicate Replace the single `manage` action with `investigate` and `communicate` per the Skills Architecture RFC. Check inspection goes under `investigate`, incident lifecycle under `communicate`. The confirmation protocol now lives in `communicate.md` where write commands are documented. Also generalizes the prepare script's reference command generation so future actions with references don't need one-off functions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update packages/cli/src/ai-context/references/communicate-incidents.md Co-authored-by: Stefan Judis <stefanjudis@gmail.com> * refactor(cli): unify action processing in prepare-ai-context Remove the special-case handling for configure and the brittle `action.id === 'configure'` guard. All actions with references now go through a single loop that applies replaceExamples uniformly (no-op when there are no template strings). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): generate skills example commands from ACTIONS The examples section in `checkly skills` was hardcoded with only initialize/configure. Generate it from the ACTIONS array so new actions like investigate and communicate appear automatically. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Stefan Judis <stefanjudis@gmail.com>
1 parent 6dbe890 commit e3fa26b

10 files changed

Lines changed: 291 additions & 33 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,27 @@ You can use the current branch of the code against any examples in the `/example
5757
2. Run `npm create checkly -- --template boilerplate-project`
5858
3. Just use `npx checkly` as normal.
5959

60+
## AI context and agent skills
61+
62+
The CLI ships agent skills (`skills/checkly/SKILL.md`) and serves reference docs via `npx checkly skills`. The source files live in `packages/cli/src/ai-context/` and are processed at build time.
63+
64+
When modifying AI context sources (`src/ai-context/skill.md`, `src/ai-context/references/`, or `src/ai-context/context.ts`):
65+
66+
1. Run the full prepare step from the `packages/cli` directory:
67+
```bash
68+
npm run prepare --workspace packages/cli
69+
```
70+
This compiles TypeScript, generates examples from fixtures, and runs `prepare-ai-context.ts` to produce the output in `dist/ai-context/`.
71+
72+
2. Copy the generated public skill to the tracked location:
73+
```bash
74+
cp packages/cli/dist/ai-context/public-skills/checkly/SKILL.md skills/checkly/SKILL.md
75+
```
76+
77+
3. Commit both your source changes and the updated `skills/checkly/SKILL.md`.
78+
79+
The `skills/checkly/SKILL.md` file is the published copy that agents load. If you forget to regenerate it, CI will flag it as out of date.
80+
6081
## Prerelease experimental version
6182

6283
To publish a NPM package for testing purpose, you can tag the pull-request with the `build` label. A GitHub Action will be

‎packages/cli/scripts/prepare-ai-context.ts‎

Lines changed: 49 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { mkdir, readFile, writeFile } from 'fs/promises'
22
import { join } from 'path'
3-
import { ACTIONS, EXAMPLE_CONFIGS, REFERENCES } from '../src/ai-context/context'
3+
import { ACTIONS, EXAMPLE_CONFIGS } from '../src/ai-context/context'
44

55
const EXAMPLES_DIR = join(__dirname, '../gen/')
66
const AI_CONTEXT_DIR = join(__dirname, '../src/ai-context')
@@ -93,10 +93,13 @@ function generateSkillCommands (): string {
9393
}).join('\n\n')
9494
}
9595

96-
function generateReferenceCommands (): string {
97-
return REFERENCES.map(ref => {
98-
const refId = ref.id.replace('configure-', '')
99-
return `### \`npx checkly skills configure ${refId}\`\n${ref.description}`
96+
function generateActionReferenceCommands (
97+
actionId: string,
98+
references: ReadonlyArray<{ id: string, description: string }>,
99+
): string {
100+
return references.map(ref => {
101+
const refId = ref.id.replace(`${actionId}-`, '')
102+
return `### \`npx checkly skills ${actionId} ${refId}\`\n${ref.description}`
100103
}).join('\n\n')
101104
}
102105

@@ -107,40 +110,56 @@ async function prepareContext () {
107110

108111
const examples = await readExampleCode()
109112

110-
// Process configure-* reference files and collect their content
111-
const referenceContents: string[] = []
112-
113-
for (const ref of REFERENCES) {
114-
const refContent = await readFile(
115-
join(AI_CONTEXT_DIR, 'references', `${ref.id}.md`),
116-
'utf8',
117-
)
118-
const processedRefContent = replaceExamples(refContent, examples)
119-
referenceContents.push(processedRefContent)
120-
121-
// Write reference file to skill output
122-
await writeOutput(processedRefContent, COMMAND_REFERENCES_DIR, `${ref.id}.md`)
113+
// Process all actions — reference files, action headers, and standalone actions
114+
const configureReferenceContents: string[] = []
115+
116+
for (const action of ACTIONS) {
117+
if ('references' in action) {
118+
for (const ref of action.references) {
119+
let refContent = await readFile(
120+
join(AI_CONTEXT_DIR, 'references', `${ref.id}.md`),
121+
'utf8',
122+
)
123+
refContent = replaceExamples(refContent, examples)
124+
await writeOutput(refContent, COMMAND_REFERENCES_DIR, `${ref.id}.md`)
125+
126+
if (action.id === 'configure') {
127+
configureReferenceContents.push(refContent)
128+
}
129+
}
130+
131+
let actionContent = await readFile(
132+
join(AI_CONTEXT_DIR, 'references', `${action.id}.md`),
133+
'utf8',
134+
)
135+
actionContent = actionContent.replace(
136+
'<!-- REFERENCE_COMMANDS -->',
137+
generateActionReferenceCommands(action.id, action.references),
138+
)
139+
actionContent = replaceExamples(actionContent, examples)
140+
await writeOutput(actionContent, COMMAND_REFERENCES_DIR, `${action.id}.md`)
141+
} else {
142+
const content = await readFile(
143+
join(AI_CONTEXT_DIR, 'references', `${action.id}.md`),
144+
'utf8',
145+
)
146+
await writeOutput(content, COMMAND_REFERENCES_DIR, `${action.id}.md`)
147+
}
123148
}
124149

125-
// Process configure.md (action header with reference links/commands + examples)
126-
let configureContent = await readFile(join(AI_CONTEXT_DIR, 'references', 'configure.md'), 'utf8')
127-
configureContent = configureContent
128-
.replace('<!-- REFERENCE_COMMANDS -->', generateReferenceCommands())
129-
configureContent = replaceExamples(configureContent, examples)
130-
await writeOutput(configureContent, COMMAND_REFERENCES_DIR, 'configure.md')
131-
132-
// Process initialize.md (no templating needed currently)
133-
const initializeContent = await readFile(join(AI_CONTEXT_DIR, 'references', 'initialize.md'), 'utf8')
134-
await writeOutput(initializeContent, COMMAND_REFERENCES_DIR, 'initialize.md')
135-
136150
// Generate SKILL.md from skill.md template with action links/commands
137151
let skillTemplate = await readFile(join(AI_CONTEXT_DIR, 'skill.md'), 'utf8')
138152
skillTemplate = skillTemplate
139153
.replace('<!-- SKILL_COMMANDS -->', generateSkillCommands())
140154
await writeOutput(skillTemplate, PUBLIC_SKILL_DIR, 'SKILL.md')
141155

142156
// Generate checkly.rules.md (configure header + all configure-* references concatenated)
143-
const demotedReferences = referenceContents.map(demoteHeadings).join('\n\n')
157+
const configureContent = await readFile(
158+
join(COMMAND_REFERENCES_DIR, 'configure.md'),
159+
'utf8',
160+
)
161+
const demotedReferences = configureReferenceContents
162+
.map(demoteHeadings).join('\n\n')
144163
const rulesContent = normalizeBlankLines(stripYamlFrontmatter(
145164
configureContent
146165
+ '\n'

‎packages/cli/src/ai-context/context.ts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,20 @@ export const REFERENCES = [
4949
},
5050
] as const
5151

52+
export const INVESTIGATE_REFERENCES = [
53+
{
54+
id: 'investigate-checks',
55+
description: 'Inspecting checks (`checks list`, `checks get`) and triggering on-demand runs',
56+
},
57+
] as const
58+
59+
export const COMMUNICATE_REFERENCES = [
60+
{
61+
id: 'communicate-incidents',
62+
description: 'Incident lifecycle (`incidents create`, `update`, `resolve`, `list`) and status pages',
63+
},
64+
] as const
65+
5266
export const SKILL = {
5367
name: 'checkly',
5468
description: 'Get all the information and context to let your agent initialize, set up, create, test and manage your monitoring checks using the Checkly CLI.',
@@ -64,6 +78,16 @@ export const ACTIONS = [
6478
description: 'Learn how to create and manage monitoring checks using Checkly constructs and the CLI.',
6579
references: REFERENCES,
6680
},
81+
{
82+
id: 'investigate',
83+
description: 'Access check status, analyze failures, and investigate errors.',
84+
references: INVESTIGATE_REFERENCES,
85+
},
86+
{
87+
id: 'communicate',
88+
description: 'Open incidents and lead customer communications via status pages.',
89+
references: COMMUNICATE_REFERENCES,
90+
},
6791
] as const
6892

6993
interface ExampleConfig {

‎packages/cli/src/ai-context/references/communicate-incidents.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Communicate Incidents
2+
3+
Create, update, and resolve incidents on status pages. All write commands require confirmation (see `npx checkly skills communicate` for the confirmation protocol).
4+
5+
## List incidents
6+
7+
```bash
8+
npx checkly incidents list
9+
npx checkly incidents list --status active --output json
10+
npx checkly incidents list --status resolved --limit 5
11+
```
12+
13+
Flags:
14+
- `--status <status>` — `active` (default), `resolved`, or `all`
15+
- `--limit <n>` — max incidents to return
16+
- `-o, --output <format>` — `table` (default), `json`, or `md`
17+
18+
## Create an incident
19+
20+
```bash
21+
npx checkly incidents create \
22+
--status-page-id <id> \
23+
--title "Service degradation" \
24+
--severity major \
25+
--message "Investigating elevated error rates" \
26+
--services <service-id-1> --services <service-id-2>
27+
```
28+
29+
Flags:
30+
- `--status-page-id <id>` — **(required)** target status page
31+
- `--title <text>` — **(required)** incident title
32+
- `--severity <level>` — `minor` (default), `medium`, `major`, or `critical`
33+
- `--message <text>` — initial update message
34+
- `--services <id>` — affected service IDs (repeat for multiple)
35+
- `--[no-]notify-subscribers` — notify status page subscribers (default: true)
36+
- `--force` — skip confirmation (required for non-interactive execution after user approval)
37+
- `--dry-run` — preview without executing
38+
39+
## Update an incident
40+
41+
```bash
42+
npx checkly incidents update <incident-id> \
43+
--message "Root cause identified" \
44+
--status identified
45+
```
46+
47+
Flags:
48+
- `--message <text>` — **(required)** update message
49+
- `--status <status>` — `investigating`, `identified`, `monitoring`, or `fixing`
50+
- `--severity <level>` — update severity
51+
- `--[no-]notify-subscribers` — notify subscribers
52+
- `--force` — skip confirmation
53+
- `--dry-run` — preview without executing
54+
55+
## Resolve an incident
56+
57+
```bash
58+
npx checkly incidents resolve <incident-id> \
59+
--message "Issue resolved, monitoring confirms recovery"
60+
```
61+
62+
Flags:
63+
- `--message <text>` — resolution message
64+
- `--[no-]notify-subscribers` — notify subscribers
65+
- `--force` — skip confirmation
66+
- `--dry-run` — preview without executing
67+
68+
## Status pages
69+
70+
Use `npx checkly status-pages list` to find status page IDs and their services. Use `--output json` to get machine-readable service IDs for incident creation.
71+
72+
```bash
73+
npx checkly status-pages list
74+
npx checkly status-pages list --output json
75+
```

‎packages/cli/src/ai-context/references/communicate.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Customer Communications and Incident Management
2+
3+
Open incidents and lead customer communications via status pages.
4+
5+
## Confirmation Protocol
6+
7+
Write commands (`incidents create`, `incidents update`, `incidents resolve`, `deploy`, `destroy`) require confirmation before executing. When the CLI detects an agent environment, it returns a JSON envelope with exit code 2 instead of executing:
8+
9+
```json
10+
{
11+
"status": "confirmation_required",
12+
"command": "incidents create",
13+
"description": "Create incident on status page",
14+
"classification": { "readOnly": false, "destructive": false, "idempotent": false },
15+
"changes": ["Will create incident \"DB outage\" on status page \"Acme\"", "Severity: major"],
16+
"confirmCommand": "checkly incidents create --title=\"DB outage\" --status-page-id=\"sp-1\" --severity=\"major\" --force"
17+
}
18+
```
19+
20+
**Rules for agents:**
21+
22+
1. When exit code is 2 and output contains `"status": "confirmation_required"`, **always present the `changes` array to the user** and ask for confirmation.
23+
2. **Never auto-append `--force`** — only run the `confirmCommand` after the user explicitly approves.
24+
3. This applies to **every** write command, not just the first one. Incident updates and resolutions also require confirmation.
25+
4. Use `--dry-run` to preview what a command will do without executing or prompting.
26+
5. Read-only commands (`incidents list`, `status-pages list`) execute immediately without confirmation.
27+
28+
## Available Commands
29+
30+
Parse and read further reference documentation when tasked with any of the following:
31+
32+
<!-- REFERENCE_COMMANDS -->

‎packages/cli/src/ai-context/references/investigate-checks.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Inspecting Checks
2+
3+
List and inspect deployed checks in your Checkly account.
4+
5+
## List checks
6+
7+
```bash
8+
npx checkly checks list
9+
npx checkly checks list --tag production --type PLAYWRIGHT
10+
npx checkly checks list --search "Homepage" --output json
11+
```
12+
13+
Flags:
14+
- `--tag <tag>` — filter by tag
15+
- `--type <type>` — filter by check type (`API`, `BROWSER`, `PLAYWRIGHT`, `MULTI_STEP`)
16+
- `--search <name>` — filter by name
17+
- `-o, --output <format>` — `table` (default), `json`, or `md`
18+
19+
## Get check details
20+
21+
```bash
22+
npx checkly checks get <check-id>
23+
npx checkly checks get <check-id> --output json
24+
```
25+
26+
Shows check configuration, recent results, and error groups.
27+
28+
### View a specific check result
29+
30+
```bash
31+
npx checkly checks get <check-id> --result <result-id>
32+
```
33+
34+
### View an error group
35+
36+
```bash
37+
npx checkly checks get <check-id> --error-group <error-group-id>
38+
```
39+
40+
## Trigger checks
41+
42+
```bash
43+
npx checkly trigger --tags production
44+
npx checkly trigger --tags critical,api
45+
```
46+
47+
Triggers existing deployed checks by tag. Useful for on-demand verification.

‎packages/cli/src/ai-context/references/investigate.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Investigating Check Status and Errors
2+
3+
Access check status, analyze failures, and investigate errors across your Checkly account.
4+
5+
Read-only commands execute immediately without confirmation.
6+
7+
## Available Commands
8+
9+
Parse and read further reference documentation when tasked with investigating any of the following:
10+
11+
<!-- REFERENCE_COMMANDS -->

‎packages/cli/src/ai-context/skill.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,10 @@ The skill is structured for efficient context usage:
2222

2323
Agents load what they need for each task.
2424

25+
## Confirmation Protocol
26+
27+
Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create.
28+
29+
Run `npx checkly skills communicate` for the full protocol details.
30+
2531
<!-- SKILL_COMMANDS -->

‎packages/cli/src/commands/skills.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,9 +68,14 @@ export default class Skills extends BaseCommand {
6868
this.log(`${SKILL.description}\n`)
6969

7070
this.log('Examples:\n')
71-
this.log(' $ checkly skills initialize')
72-
this.log(' $ checkly skills configure')
73-
this.log(' $ checkly skills configure api-checks')
71+
for (const action of ACTIONS) {
72+
this.log(` $ checkly skills ${action.id}`)
73+
if ('references' in action) {
74+
const firstRef = action.references[0]
75+
const refId = firstRef.id.replace(`${action.id}-`, '')
76+
this.log(` $ checkly skills ${action.id} ${refId}`)
77+
}
78+
}
7479
this.log('')
7580

7681
this.log('Actions:\n')

0 commit comments

Comments
 (0)