feat(cli): add AgenticCheck construct by thebiglabasky · Pull Request #1268 · checkly/checkly-cli

thebiglabasky · 2026-03-20T16:10:14Z

Summary

Adds AgenticCheck construct for defining AI-powered monitoring checks via checkly.config.ts
Extends Check (no runtime/bundling needed) with a prompt field (required, max 10,000 chars)
Validates frequency minimum of 30 minutes (agentic checks are expensive)
Adds AGENTIC to CheckTypes enum and analytics mappings

Usage

import { AgenticCheck } from 'checkly/constructs'

new AgenticCheck('homepage-health', {
  name: 'Homepage Health Check',
  prompt: 'Navigate to https://example.com and verify the page loads correctly.',
  frequency: 30,
  locations: ['eu-west-1'],
})

Backend dependency

Requires the companion backend PR to update the CLI deploy schema: https://github.com/checkly/monorepo/pull/1098

Test plan

6 unit tests: basic synthesis, config defaults, group mapping, prompt validation (empty/too long), frequency validation
TypeScript compiles cleanly
Tested locally with checkly deploy --preview against local backend

🤖 Generated with Claude Code

MichaelHogers · 2026-03-23T08:58:09Z

@thebiglabasky 🔥

Adds the AgenticCheck construct that lets users define agentic checks in their Checkly project via checkly.config.ts. Agentic checks use a prompt string (no scripts/bundling) to define what the AI agent should verify. - Add AGENTIC to CheckTypes enum - Create AgenticCheck extending Check (not RuntimeCheck) - Validate prompt (required, max 10000 chars) and frequency (min 30) - Add construct export and analytics mapping - Add 6 tests covering synthesis, defaults, groups, and validation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Agentic checks always run in parallel for better reporting UX. The construct omits runParallel from props (users can't override it) and hardcodes it to true in synthesize(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This reverts commit a77f117.

Lock down AgenticCheckProps to only the subset of CheckProps the Checkly platform currently honors for agentic checks. Omits locations, privateLocations, runParallel, retryStrategy, shouldFail, doubleCheck, triggerIncident and groupId at the type level, and restricts frequency to the discrete set exposed in the webapp builder. Defensive overrides in the constructor ensure project-level config defaults for any of these fields are also ignored, so the construct never claims to support what the platform won't honor. These restrictions can be relaxed additively (without breaking changes) once the platform supports the corresponding capabilities for agentic checks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…cess Adds an `agentRuntime` prop to AgenticCheck that declares the runtime context the agent has access to during execution: which skills it can use and which environment variables it is permitted to read. agentRuntime is the explicit security boundary for the agent. Anything not declared here is unavailable at runtime, which keeps the blast radius of any prompt injection as small as possible. Environment variables accept either bare names or `{ name, description }` objects; descriptions are passed to the agent so it can decide when to read each variable, and are validated to stay within the runner's 200-char limit. The construct always synthesizes a complete agentRuntime payload (with empty arrays when unset), so the backend has unambiguous source-of-truth semantics: omitted skills/env vars mean "the agent should not have them", not "preserve whatever was there before". The backend translates this clean shape into the existing agenticCheckData storage blob. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

sorccu

Please include codegen (for import) unless not feasible.

sorccu · 2026-04-06T16:38:12Z

These are the required steps when creating a new construct:

Add construct
Add assertion builder (if relevant)
Add code generation support for the new construct
Add tests
Add examples
Include the new construct in E2E tests
Update the reporter for the new construct
Update AI context skill templates

Can you make sure these are all done?

Wires AgenticCheck into the \`checkly import\` code generation flow so users can import an existing agentic check from the backend and get a construct file that matches the locked-down props type added earlier on this branch. Implementation: - New \`AgenticCheckCodegen\` in \`agentic-check-codegen.ts\` mirroring the Heartbeat/Browser codegen pattern. Places generated files under \`resources/agentic-checks/{name}\`, imports \`AgenticCheck\` from \`checkly/constructs\`, and emits a \`new AgenticCheck(...)\` expression. - The resource shape (\`AgenticCheckResource\`) carries \`prompt\` and the backend's \`agenticCheckData\` storage blob. A small helper reverse- translates that blob into the CLI construct's public shape: * \`agenticCheckData.skills\` → \`agentRuntime.skills\` * \`agenticCheckData.selectedEnvironmentVariables\` → \`agentRuntime.environmentVariables\` — mapping \`key\` back to \`name\` for object-form entries * \`agenticCheckData.assertionRules\` → deliberately not emitted (agent-generated, preserved server-side via the backend's merge logic, the construct intentionally does not expose them) An imported check with no skills and no selected env vars produces no \`agentRuntime\` block at all, to avoid noise in the generated code. - Registers the new codegen in \`CheckCodegen.constructor\` and wires \`AGENTIC\` into the dispatch switches in \`describe()\` and \`gencode()\` alongside the existing check types. \`buildCheckProps\` gains a narrow \`BuildCheckPropsOptions\` parameter — currently a single \`skipRetryStrategy\` flag — because \`retryStrategy\` is the only field that \`buildCheckProps\` emits unconditionally (null is rendered as \`RetryStrategyBuilder.noRetries()\`). The AgenticCheck construct's props type omits \`retryStrategy\` entirely, so emitting it would produce a generated file that does not type-check. The AgenticCheckCodegen passes \`skipRetryStrategy: true\` and clears \`locations\` from a resource copy before calling \`buildCheckProps\` — the two fixes together make the generated output round-trip cleanly through the construct's props type. Verified end-to-end by extending \`ai-context/context.fixtures.json\` with an example agentic check (prompt, skills, mixed-form env vars, sample assertionRules) and running \`npm run prepare:ai-context\`. The generated \`gen/resources/agentic-checks/example-agentic-check.check.ts\` contains only fields the construct accepts, reverse-translates env vars from \`key\` to \`name\`, omits \`assertionRules\`, and type-checks against the construct. Full CLI suite remains green (829 passing / 2 skipped / 60 files). No regressions in any other check type's codegen from the new optional options parameter. Covers item #3 on Simo's new-construct checklist. Items #5 (examples), #6 (E2E tests), and #8 (AI context reference doc) still pending on this branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

thebiglabasky · 2026-04-06T21:04:35Z

Update the reporter for the new construct

Requires some API updates so we pass down check results in a similar way as we do for other checks: https://github.com/checkly/monorepo/pull/1313
The CLI PR that digests this and updates reporters (can be merged separately from the construct PR): #1277

I'll update this one with what is directly dependent on the construct

Cover the codegen edge cases that the `prepare:ai-context` happy-path fixture doesn't exercise: missing/null/empty `agenticCheckData`, skills-only, env vars-only, mixed bare-string and object-form env var entries, the `key` -> `name` reverse translation, and the explicit suppression of `locations`, `retryStrategy`, and `assertionRules` from the generated source. Also drop a few defensive runtime checks in the codegen that the typed `StoredAgenticCheckData` interface already guarantees, and trim a comment block in `gencode()`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add an `AGENTIC_CHECK` entry to `EXAMPLE_CONFIGS` so the `prepare:ai-context` pipeline can interpolate the generated example into reference docs. - Simplify the `example-agentic-check` fixture to a single self-contained pricing-page check (no skills, no env vars). The unit tests added in the previous commit cover the more complex agentRuntime cases. - Add an `agentic.check.ts` (and the JS twin) to the boilerplate project so first-time CLI users see what an agentic check looks like alongside the existing API/URL/heartbeat examples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds a `deploy-agentic-project` fixture that exercises both the plain-prompt form and the `agentRuntime` form, plus a corresponding `describe('deploy-agentic-project', ...)` block in `deploy.spec.ts`. The tests cover two scenarios: 1. `--preview` produces an import plan that lists both agentic checks under the `Create:` heading, exercising the codegen + deploy planner path without leaving any state behind. 2. `--force` actually creates the checks on the target account and reads them back via the public REST API to confirm that `checkType`, the hardcoded `us-east-1` location, and round-tripped tags all land correctly. Both tests require the e2e CI account to be entitled to agentic checks. If the account is not entitled, the tests will fail with a 402 — that's the signal to enable the entitlement, not to weaken the test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add `configure-agentic-checks` to the `REFERENCES` array so the `npx checkly skills` command and the generated `checkly.rules.md` expose the new construct alongside the other check types. - Add `configure-agentic-checks.md` covering the locked-down props, the restricted frequency set, the `agentRuntime` security boundary, the prompt-writing guidance, and the assertion-rules round-trip rules. The example is interpolated from `EXAMPLE_CONFIGS.AGENTIC_CHECK`. - Mention agentic checks in the `skill.md` frontmatter description so agents discover the construct from the skill summary alone. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions · 2026-04-06T21:51:33Z

⚠️ AI context is out of date

The skills/ directory doesn't match the generated output from packages/cli.

Please run the following locally and commit the changes:

npm run prepare

Skills declared on `agentRuntime.skills` are installed on the runner via `npx skills add <entry>` (https://skills.sh), not via the `npx checkly skills install` meta-skill installer that ships the SKILL.md file into a project. The previous JSDoc, reference doc, and example fixtures conflated the two. Updates: - `agentic-check.ts`: rewrite the `AgentRuntime.skills` JSDoc to describe the real installation path, list the three identifier forms skills.sh accepts (full URL, owner/repo, plain name), and note that the runner preloads the `playwright-cli` skill so users only need to declare extras. - `configure-agentic-checks.md`: same correction in the AI reference doc, with the example block expanded to show all three forms. - Test/example fixtures: switch the placeholder skill from the non-existent `'checkly/playwright-skill'` to `'addyosmani/web-quality-skills'` to make it visible that third-party skills published to skills.sh work — not just Checkly's own. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds a debug step before `npm run test:e2e` (ubuntu only) that prints SHA-256 hashes of the secret account ID/name plus a few non-secret fields from `/next/accounts/<id>` and `/v1/accounts/me/entitlements`. The hashes let us verify which account the secrets resolve to without leaking the actual values in this public repo's CI logs. The entitlement block shows whether `maxAgenticChecks` bumps are landing on the account the e2e tests actually use, and the existing-AGENTIC count surfaces stale orphans that may be eating the budget. Compare locally with: node -e "console.log(require('crypto').createHash('sha256') \\ .update('<candidate-id-or-name>').digest('hex'))" To remove once the right account is confirmed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The fingerprint did its job — it surfaced that the secrets pointed at an account different from the one being bumped, and that the target account's `maxAgenticChecks` column was unset (the `add_agentic_check_quota_columns` migration only seeded HOBBY, STARTER, TEAM and TRIAL plan accounts, leaving CONTRACT plan rows with NULL). The right account has been bumped and the e2e tests should pass on the next run. Reverting the workflow back to its previous shape. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

thebiglabasky and others added 5 commits April 6, 2026 12:02

Revert "feat(cli): force runParallel for agentic checks"

5a24001

This reverts commit a77f117.

thebiglabasky force-pushed the herve/AI-114/agentic-check-cli-construct branch from 9e02521 to e24774d Compare April 6, 2026 12:29

sorccu requested changes Apr 6, 2026

View reviewed changes

thebiglabasky and others added 4 commits April 6, 2026 23:33

thebiglabasky and others added 6 commits April 7, 2026 08:09

fix: regen ai context and fix NO_RETRIES

c8c922e

Merge branch 'main' into herve/AI-114/agentic-check-cli-construct

c8e1088

Merge branch 'main' into herve/AI-114/agentic-check-cli-construct

4768324

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): add AgenticCheck construct#1268

feat(cli): add AgenticCheck construct#1268
thebiglabasky wants to merge 16 commits intomainfrom
herve/AI-114/agentic-check-cli-construct

thebiglabasky commented Mar 20, 2026 •

edited

Loading

Uh oh!

MichaelHogers commented Mar 23, 2026

Uh oh!

sorccu left a comment

Uh oh!

sorccu commented Apr 6, 2026

Uh oh!

thebiglabasky commented Apr 6, 2026

Uh oh!

github-actions bot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

thebiglabasky commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Backend dependency

Test plan

Uh oh!

MichaelHogers commented Mar 23, 2026

Uh oh!

sorccu left a comment

Choose a reason for hiding this comment

Uh oh!

sorccu commented Apr 6, 2026

Uh oh!

thebiglabasky commented Apr 6, 2026

Uh oh!

github-actions bot commented Apr 6, 2026

⚠️ AI context is out of date

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thebiglabasky commented Mar 20, 2026 •

edited

Loading