fix: adjustments on agentic check construct in prep for self-service billing rollout (#1293)

* fix: adjustments on agentic check construct in prep for self-service billing rollout

* fix: remove agentic env exposure from CLI

Author: thebiglabasky

examples/boilerplate-project-js/__checks__/agentic.check.js

@@ -8,8 +8,7 @@ new AgenticCheck('checkly-pricing-page', {
   name: 'Checkly pricing page',
   prompt:
     'Navigate to https://www.checklyhq.com/pricing and verify that at least three plan tiers are displayed on the page.',
-  // Agentic checks currently run from a single location and follow their own
-  // frequency cadence (30, 60, 120, 180, 360, 720 or 1440 minutes). The
-  // construct hardcodes the location and validates the frequency for you.
-  frequency: 60,
+  // The backend enforces the allowed cadence and region count for your account.
+  frequency: 5,
+  locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'],
 })

examples/boilerplate-project/__checks__/agentic.check.ts

@@ -8,8 +8,7 @@ new AgenticCheck('checkly-pricing-page', {
   name: 'Checkly pricing page',
   prompt:
     'Navigate to https://www.checklyhq.com/pricing and verify that at least three plan tiers are displayed on the page.',
-  // Agentic checks currently run from a single location and follow their own
-  // frequency cadence (30, 60, 120, 180, 360, 720 or 1440 minutes). The
-  // construct hardcodes the location and validates the frequency for you.
-  frequency: 60,
+  // The backend enforces the allowed cadence and region count for your account.
+  frequency: 5,
+  locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'],
 })

packages/cli/e2e/__tests__/deploy.spec.ts

@@ -314,7 +314,7 @@ describe('deploy', { timeout: 45_000 }, () => {
 
       expect(pricingCheck).toBeDefined()
       expect(pricingCheck.checkType).toEqual('AGENTIC')
-      // The construct hardcodes a single location for agentic checks.
+      // The construct keeps a us-east-1 fallback when no location is configured.
       expect(pricingCheck.locations).toEqual(['us-east-1'])
       expect(pricingCheck.tags).toEqual(expect.arrayContaining(['e2e', 'agentic']))
 

packages/cli/e2e/__tests__/fixtures/deploy-agentic-project/agentic.check.ts

@@ -20,9 +20,5 @@ new AgenticCheck('agentic-runtime-check', {
   frequency: 60,
   agentRuntime: {
     skills: ['addyosmani/web-quality-skills'],
-    exposeEnvironmentVariables: [
-      'ENVIRONMENT_URL',
-      { name: 'TEST_USER_EMAIL', description: 'Login email for the test account' },
-    ],
   },
 })

packages/cli/src/ai-context/context.fixtures.json

@@ -834,13 +834,15 @@
           "id": "dddddddd-dddd-dddd-dddd-dddddddddddd",
           "checkType": "AGENTIC",
           "name": "Example Agentic Check",
-          "frequency": 60,
+          "frequency": 5,
           "frequencyOffset": 0,
           "activated": true,
           "muted": false,
           "shouldFail": false,
           "locations": [
-            "us-east-1"
+            "us-east-1",
+            "eu-west-1",
+            "ap-southeast-1"
           ],
           "accountId": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
           "created_at": "2025-10-28T17:25:17.037Z",
@@ -874,7 +876,6 @@
           "prompt": "Navigate to https://www.checklyhq.com/pricing and verify that at least three plan tiers are displayed on the page.",
           "agenticCheckData": {
             "skills": [],
-            "selectedEnvironmentVariables": [],
             "assertionRules": []
           },
           "alertChannelSubscriptions": [],

packages/cli/src/ai-context/references/configure-agentic-checks.md

@@ -4,15 +4,15 @@
 - Agentic checks are AI-powered: instead of writing code, you describe what the check should do in natural language with the `prompt` property. The agent decides how to satisfy the prompt at runtime.
 - Write prompts as concrete imperative steps, not vague goals. Tell the agent which URL to navigate to and what specific signals confirm success — for example, "Navigate to https://example.com/pricing and verify that at least three plan tiers are displayed", not "Check that pricing works".
 - Keep prompts under 10000 characters. The construct will fail validation otherwise.
-- **Frequency is restricted.** Only `30`, `60`, `120`, `180`, `360`, `720`, or `1440` minutes are accepted (matching `Frequency.EVERY_30M`, `EVERY_1H`, `EVERY_2H`, `EVERY_3H`, `EVERY_6H`, `EVERY_12H`, `EVERY_24H`). Anything else fails validation.
-- **Locations are not configurable.** Agentic checks currently run from a single fixed location. The construct hardcodes it — do not pass `locations` or `privateLocations`.
+- **Frequency is entitlement-gated by the backend.** You can configure the same `frequency` values as other checks, including `Frequency.EVERY_5M` / `5`. The backend enforces the fastest cadence available to the account.
+- **Locations are entitlement-gated by the backend.** You can pass `locations` just like other public checks. The backend enforces the allowed number of regions for the account. `privateLocations` are not supported.
 - **Several common check fields are intentionally omitted** from `AgenticCheckProps`: `runParallel`, `retryStrategy`, `shouldFail`, `doubleCheck`, `triggerIncident`, and `groupId`. The platform does not yet honor these for agentic checks. Setting them in the construct is a TypeScript error.
 - **Important:** The target URL must be publicly accessible. Checks run on Checkly's cloud infrastructure, not locally. If the user is developing against localhost, suggest a tunneling tool (ngrok, cloudflare tunnel) or a preview/staging deployment.
 - **Plan-gated:** Agentic checks require the `AGENTIC_CHECKS` entitlement on the account. Run `npx checkly skills manage` to check entitlements before using.
 
-## `agentRuntime` — security boundary for skills and env vars
+## `agentRuntime` skills
 
-`agentRuntime` is the explicit allowlist of resources the agent may use at execution time. Anything not declared in `agentRuntime` is **unavailable** to the agent. Treat it as a security boundary: the smaller the runtime surface, the smaller the blast radius of any prompt injection.
+`agentRuntime` is used to add skills the agent may use at execution time.
 
 ```typescript
 agentRuntime: {
@@ -25,23 +25,10 @@ agentRuntime: {
   //   - owner/repo form: 'addyosmani/web-quality-skills'
   //   - plain name:      'cost-optimization'
   skills: ['addyosmani/web-quality-skills'],
-
-  // Environment variables the agent is allowed to read at runtime.
-  // Anything not listed here is hidden from the agent process — even
-  // if it's defined at the project or check level.
-  exposeEnvironmentVariables: [
-    // Bare string form: variable name only.
-    'ENVIRONMENT_URL',
-    // Object form: pair the variable with a description so the agent
-    // can decide when to read it. Descriptions are passed to the model
-    // and are truncated to 200 characters.
-    { name: 'TEST_USER_EMAIL', description: 'Login email for the test account' },
-  ],
 },
 ```
 
-- Only declare env vars the agent **needs**. Adding a variable to `exposeEnvironmentVariables` exposes it to the model and to anything the model invokes via skills.
-- Descriptions are not just documentation — they steer the model's decisions. Use them to disambiguate variables that have non-obvious names.
+- Reference Checkly environment variables directly in the prompt with double brackets, for example `{{ENVIRONMENT_URL}}`.
 - The runner installs each skill via `npx skills add` at the start of every check run. The CLI does not validate the skill identifier at deploy time, so a typo will not surface until the first run.
 - The `playwright-cli` skill is preloaded for every agentic check. Only declare additional skills here.
 

packages/cli/src/constructs/__tests__/agentic-check-codegen.spec.ts

@@ -88,15 +88,14 @@ describe('AgenticCheckCodegen', () => {
       expect(source).toContain('prompt: \'Verify the homepage loads.\'')
     })
 
-    it('should not emit `locations` even when the backend returns one', async () => {
-      // The backend forces a single location for agentic checks today and
-      // the construct hardcodes it. Surfacing it in generated code would
-      // break type-checking against `AgenticCheckProps`.
+    it('should emit `locations` when the backend returns them', async () => {
       const source = await renderResource(env, baseResource({
-        locations: ['us-east-1'],
+        locations: ['us-east-1', 'eu-west-1'],
       }))
 
-      expect(source).not.toContain('locations')
+      expect(source).toContain('locations: [')
+      expect(source).toContain('\'us-east-1\'')
+      expect(source).toContain('\'eu-west-1\'')
     })
 
     it('should not emit `retryStrategy`', async () => {
@@ -127,11 +126,10 @@ describe('AgenticCheckCodegen', () => {
       expect(source).not.toContain('agentRuntime')
     })
 
-    it('should not emit `agentRuntime` when skills and env vars are both empty', async () => {
+    it('should not emit `agentRuntime` when skills are empty', async () => {
       const source = await renderResource(env, baseResource({
         agenticCheckData: {
           skills: [],
-          selectedEnvironmentVariables: [],
         },
       }))
       expect(source).not.toContain('agentRuntime')
@@ -148,8 +146,6 @@ describe('AgenticCheckCodegen', () => {
       expect(source).toContain('skills: [')
       expect(source).toContain('\'addyosmani/web-quality-skills\'')
       expect(source).toContain('\'cost-optimization\'')
-      // No empty exposeEnvironmentVariables when none are selected.
-      expect(source).not.toContain('exposeEnvironmentVariables')
     })
 
     it('should drop empty skill names', async () => {
@@ -165,67 +161,6 @@ describe('AgenticCheckCodegen', () => {
       expect(source).not.toMatch(/skills:\s*\[[^\]]*''/m)
     })
 
-    it('should emit `agentRuntime.exposeEnvironmentVariables` for bare-string entries', async () => {
-      const source = await renderResource(env, baseResource({
-        agenticCheckData: {
-          selectedEnvironmentVariables: ['ENVIRONMENT_URL', 'API_KEY'],
-        },
-      }))
-
-      expect(source).toContain('exposeEnvironmentVariables: [')
-      expect(source).toContain('\'ENVIRONMENT_URL\'')
-      expect(source).toContain('\'API_KEY\'')
-      // Bare strings should not be expanded into objects. The only `name:`
-      // we expect in the file is the top-level `name: 'Agentic Check'`.
-      expect((source.match(/name:/g) ?? []).length).toEqual(1)
-    })
-
-    it('should reverse-translate `key` into `name` for object-form entries', async () => {
-      // The backend stores selected env vars as `{ key, description? }` but
-      // the construct exposes `{ name, description? }`. The codegen has to
-      // translate on the way out so the generated file matches the construct
-      // type.
-      const source = await renderResource(env, baseResource({
-        agenticCheckData: {
-          selectedEnvironmentVariables: [
-            { key: 'TEST_USER_EMAIL', description: 'Login email for the test account' },
-          ],
-        },
-      }))
-
-      expect(source).toContain('name: \'TEST_USER_EMAIL\'')
-      expect(source).toContain('description: \'Login email for the test account\'')
-      expect(source).not.toContain('key:')
-    })
-
-    it('should omit `description` when not provided on object-form entries', async () => {
-      const source = await renderResource(env, baseResource({
-        agenticCheckData: {
-          selectedEnvironmentVariables: [
-            { key: 'PLAIN_KEY' },
-          ],
-        },
-      }))
-
-      expect(source).toContain('name: \'PLAIN_KEY\'')
-      expect(source).not.toContain('description')
-    })
-
-    it('should support a mix of bare-string and object-form env vars', async () => {
-      const source = await renderResource(env, baseResource({
-        agenticCheckData: {
-          selectedEnvironmentVariables: [
-            'ENVIRONMENT_URL',
-            { key: 'TEST_USER_EMAIL', description: 'Login email' },
-          ],
-        },
-      }))
-
-      expect(source).toContain('\'ENVIRONMENT_URL\'')
-      expect(source).toContain('name: \'TEST_USER_EMAIL\'')
-      expect(source).toContain('description: \'Login email\'')
-    })
-
     it('should never emit `assertionRules`', async () => {
       // Assertion rules are agent-generated and preserved server-side. The
       // construct does not accept them, so generated code must not surface

packages/cli/src/constructs/__tests__/agentic-check.spec.ts

@@ -69,7 +69,6 @@ describe('AgenticCheck', () => {
               runParallel: false,
               agentRuntime: {
                 skills: [],
-                exposeEnvironmentVariables: [],
               },
             }),
           }),
@@ -78,7 +77,7 @@ describe('AgenticCheck', () => {
     }))
   }, DEFAULT_TEST_TIMEOUT)
 
-  it('should expose agentRuntime skills and environment variables', async () => {
+  it('should expose agentRuntime skills', async () => {
     const output = await parseProject(
       fixt,
       '--config',
@@ -99,10 +98,6 @@ describe('AgenticCheck', () => {
               checkType: 'AGENTIC',
               agentRuntime: {
                 skills: ['addyosmani/web-quality-skills'],
-                exposeEnvironmentVariables: [
-                  'API_KEY',
-                  { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' },
-                ],
               },
             }),
           }),
@@ -114,7 +109,6 @@ describe('AgenticCheck', () => {
               checkType: 'AGENTIC',
               agentRuntime: {
                 skills: [],
-                exposeEnvironmentVariables: [],
               },
             }),
           }),
@@ -123,7 +117,7 @@ describe('AgenticCheck', () => {
     }))
   }, DEFAULT_TEST_TIMEOUT)
 
-  it('should default to a single us-east-1 location and ignore project-level locations', async () => {
+  it('should apply project-level locations', async () => {
     const output = await parseProject(
       fixt,
       '--config',
@@ -142,7 +136,36 @@ describe('AgenticCheck', () => {
             member: true,
             payload: expect.objectContaining({
               checkType: 'AGENTIC',
-              locations: ['us-east-1'],
+              locations: ['eu-west-1', 'ap-southeast-1'],
+              frequency: 30,
+              runParallel: false,
+            }),
+          }),
+        ]),
+      }),
+    }))
+  }, DEFAULT_TEST_TIMEOUT)
+
+  it('should apply check-level locations', async () => {
+    const output = await parseProject(
+      fixt,
+      '--config',
+      fixt.abspath('test-cases/test-locations-explicit/checkly.config.js'),
+    )
+
+    expect(output).toEqual(expect.objectContaining({
+      diagnostics: expect.objectContaining({
+        fatal: false,
+      }),
+      payload: expect.objectContaining({
+        resources: expect.arrayContaining([
+          expect.objectContaining({
+            logicalId: 'locations-explicit',
+            type: 'check',
+            member: true,
+            payload: expect.objectContaining({
+              checkType: 'AGENTIC',
+              locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'],
               frequency: 30,
               runParallel: false,
             }),
@@ -257,7 +280,7 @@ describe('AgenticCheck', () => {
     }))
   }, DEFAULT_TEST_TIMEOUT)
 
-  it('should fail validation when frequency is not in the supported set', async () => {
+  it('should allow faster frequencies so the backend can enforce account entitlements', async () => {
     const output = await parseProject(
       fixt,
       '--config',
@@ -266,48 +289,19 @@ describe('AgenticCheck', () => {
 
     expect(output).toEqual(expect.objectContaining({
       diagnostics: expect.objectContaining({
-        fatal: true,
-        observations: expect.arrayContaining([
-          expect.objectContaining({
-            message: expect.stringContaining('"frequency" must be one of 30, 60, 120, 180, 360, 720, 1440'),
-          }),
-        ]),
-      }),
-    }))
-  }, DEFAULT_TEST_TIMEOUT)
-
-  it('should fail validation when an environment variable name is empty', async () => {
-    const output = await parseProject(
-      fixt,
-      '--config',
-      fixt.abspath('test-cases/test-validation-empty-env-var-name/checkly.config.js'),
-    )
-
-    expect(output).toEqual(expect.objectContaining({
-      diagnostics: expect.objectContaining({
-        fatal: true,
-        observations: expect.arrayContaining([
-          expect.objectContaining({
-            message: expect.stringContaining('"agentRuntime.exposeEnvironmentVariables[0]" must have a non-empty name'),
-          }),
-        ]),
+        fatal: false,
       }),
-    }))
-  }, DEFAULT_TEST_TIMEOUT)
-
-  it('should fail validation when an environment variable description exceeds 200 characters', async () => {
-    const output = await parseProject(
-      fixt,
-      '--config',
-      fixt.abspath('test-cases/test-validation-description-too-long/checkly.config.js'),
-    )
-
-    expect(output).toEqual(expect.objectContaining({
-      diagnostics: expect.objectContaining({
-        fatal: true,
-        observations: expect.arrayContaining([
+      payload: expect.objectContaining({
+        resources: expect.arrayContaining([
           expect.objectContaining({
-            message: expect.stringContaining('"agentRuntime.exposeEnvironmentVariables[0].description" must be at most 200 characters'),
+            logicalId: 'low-frequency',
+            type: 'check',
+            member: true,
+            payload: expect.objectContaining({
+              checkType: 'AGENTIC',
+              frequency: 5,
+              locations: ['us-east-1'],
+            }),
           }),
         ]),
       }),

packages/cli/src/constructs/__tests__/fixtures/agentic-check/test-cases/test-agent-runtime/test.check.js

@@ -5,10 +5,6 @@ new AgenticCheck('agent-runtime-check', {
   prompt: 'Sign in to the test account and verify the dashboard loads.',
   agentRuntime: {
     skills: ['addyosmani/web-quality-skills'],
-    exposeEnvironmentVariables: [
-      'API_KEY',
-      { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' },
-    ],
   },
 })