feat(ssl): property-scoped assertion grammar

[danielpaulus]

What

Redesigns the SSL monitor assertion grammar in the CLI constructs to a property-scoped model matching the backend's assertion grammar, and reworks the assertion builders behind it into a single-declaration grammar system that SSL and traceroute now share.

Public API (SSL)

SslAssertionBuilder is now certificate(property), connection(property), responseTime(), jsonResponse(jsonPath), textResponse(regex?) — replacing the flat per-source methods (certExpiresInDays(), tlsVersion(), sanContains(), …).

• Each property exposes only the operators the backend accepts for it, with its target typed: numeric properties take a number, booleans a boolean, and tlsVersion / signatureAlgorithm their constant-map unions (TlsVersion / SignatureAlgorithm / CipherSuite are kept as targets). So certificate('daysUntilExpiry').greaterThan('30 days') and hopCount().notEquals(5) are compile errors rather than assertions that silently never fire.
• The SSL-only GREATER_THAN_OR_EQUAL and MATCHES operators are dropped (along with the now-unused valueForBooleanAssertion / hasMatches codegen path — SSL was its sole caller; other check types verified unaffected).
• textResponse(pattern) now carries the pattern in the assertion's property slot — the slot the backend reads the extraction pattern from. It previously went into regex, which is never consulted, so the pattern was silently ignored and the comparison ran against the whole response document.

Breaking: SslMonitor shipped in 8.15/8.16 with the flat methods; this removes them. The bundled create-checkly templates are migrated in the same PR so scaffolding keeps working (see Other changes).

Internal: single-declaration grammar system

Each assertion property is declared once, as data — its operators and its target type — in internal/assertion-grammar.ts. The typed builder surface, the runtime validation whitelist, and codegen's literal kind are all derived from that one declaration, so a property cannot expose an operator its whitelist rejects, or a target type codegen renders wrong. This replaces four hand-synced declaration sites (per-property classes, an operator map, a value-type table, a comparison whitelist) that a prior review showed could drift while compiling clean.

• SSL (ssl-assertion-grammar.ts) and traceroute (traceroute-assertion-grammar.ts) are both built on it; traceroute's builder classes and hand-maintained validation tables are removed with no behavior change.
• Consistency specs iterate each grammar and assert the builder, validation, and codegen agree for every declared (source, property, operator).

Validation

Assertions written as plain object literals bypass the builder and are type-legal, so they are checked at deploy: unknown source / property / unsupported comparison, and — new — a non-numeric target on a numeric property or a non-true/false target on a boolean property, closing a silent-never-fires hole on that path.

Other changes

• create-checkly advanced-project (TS) and advanced-project-js (JS) templates ported to the new grammar — the recommended template pins checkly: latest, so without this the scaffolded project would fail to load its SSL check on release.
• AI-context SSL example updated to the new grammar.

Tests

tsc --build clean, eslint clean, full CLI unit suite green. New consistency specs for SSL and traceroute; existing SSL and traceroute tests pass unmodified (wire output byte-identical, operator/target narrowing verified via compile-time checks).

Known follow-up

Spec files are not type-checked by the build (RED-739), so the @ts-expect-error narrowing assertions in the specs are inert until that lands; the grammar's guarantees hold at tsc on src and are additionally covered by the runtime consistency specs.

🤖 Generated with Claude Code

[sorccu]

Review: property-scoped assertion grammar

Verdict: request changes. The core refactor is well-executed, but it misses a call site that ships the removed API to every new user via the recommended create-checkly template. Two further issues undermine guarantees the PR believes it has. Each finding below was verified by executing code, not just reading it.

What's good

The design is sound and the alignment with the API-check grammar is the right call. Reusing GeneralAssertionBuilder/NumericAssertionBuilder rather than hand-rolling 12 builder classes is a genuine simplification (-494 lines), and dropping the boolean codegen path is correctly scoped — SSL was its only caller. The 46 SSL tests pass and tsc --build is clean.

Blocker

The create-checkly templates still call the deleted methods. examples/advanced-project/src/__checks__/uptime/ssl.check.ts:21-24 and examples/advanced-project-js/.../ssl.check.js:21-24 still use certExpiresInDays(), chainTrusted(), hostnameVerified(), and tlsVersion().

These aren't inert docs:

• packages/create-cli/src/utils/installation.ts:69 downloads them as github:checkly/checkly-cli/examples/${template}#${version}
• packages/create-cli/src/utils/prompts.ts:8 labels advanced-project "(recommended)"
• packages/create-cli/src/actions/dependencies.ts:16 force-pins checkly: 'latest'

So the moment a release containing this PR is cut, npm create checkly → recommended template → checkly deploy dies with TypeError: SslAssertionBuilder.certExpiresInDays is not a function before the user writes a line of their own code. The version skew is guaranteed, not incidental: old-grammar template from the git tag, new-grammar CLI from latest. The TS template also fails tsc; the JS one fails purely at runtime.

The identical example inside ai-context/context.ts was updated — this looks like a missed second copy. CI can't catch it: examples/ is outside the pnpm workspace (pnpm-workspace.yaml lists only packages/*), ESLint ignores examples/*, and no workflow references it. The repo's own checkly-cli-new-monitor checklist already names both files as part of monitor work.

Suggested fix: port both files to certificate('daysUntilExpiry').greaterThan(30) / connection('chainTrusted').equals(true) / connection('hostnameVerified').equals(true) / connection('tlsVersion').equals('TLS1.3') in this PR, and consider a create-cli e2e assertion that actually parses the scaffolded project so template drift fails CI.

Major

The type narrowing (compile-time) tests assert nothing. The block's new comment claims "tsc --build fails if any becomes a false positive (an unused expect-error)." That is false:

• packages/cli/tsconfig.json sets "exclude": ["src/**/__tests__", ...], and prepare:dist is tsc --build against that config
• vitest.config.mts declares no test.typecheck block, so specs are transpiled by esbuild with types dropped
• eslint.config.mjs sets @typescript-eslint/ban-ts-comment: 0 and isn't type-aware

Reproduced: appending a deliberately-unused @ts-expect-error to the spec passes both tsc --build and eslint clean. So the narrowing deliberately restored by 12ab12f has zero regression coverage — deleting the signatureAlgorithm/tlsVersion overloads ships green, and the same hole covers the certificate('bogusProperty') narrowing that is the stated reason for the property unions.

Either wire up test.typecheck against a tests-inclusive tsconfig, or correct the comment so nobody relies on it.

The validation whitelists can drift from the builder unions silently. certificateProperties (ssl-assertion-validation.ts:37) and connectionProperties (:52) are typed Record<string, PropertyRule>, with no compile-time link to CertificateProperty/ConnectionProperty.

This contradicts the file's own source-level rule one block down (Record<SslAssertion['source'], SslSourceRule>, commented "Keyed by the source union so adding a source without a rule fails to compile") and the sibling traceroute-assertion-validation.ts, which types its list as Record<TracerouteResponseTimeProperty, true> for exactly this reason. The protection is applied where there are 5 entries and dropped where there are 19.

Verified: adding 'notAfter' to CertificateProperty alone compiles clean with zero diagnostics and autocompletes, then fails at the customer's checkly deploy with "has an unknown property". The comment at ssl-assertion.ts:107-110 ("The runtime whitelist in ssl-assertion-validation.ts is the source of truth") actively encourages readers to assume the two are linked when nothing enforces it. Fix is one line each once the unions are exported.

Minor

• Numeric properties lost their number contract. certificate('daysUntilExpiry') and certificate('keySizeBits') fall through to the untyped overload (TargetType = string | number | boolean), so .greaterThan('30 days') compiles, deploys, and silently never fires — GREATER_THAN is whitelisted and target values are only checked when booleanTarget is set. NumericAssertionBuilder<Source, Property> already accepts a property and exposes exactly the four NUMBER operators, and the per-property overload machinery already exists in this method. Given the PR keeps narrowing for the typed constants, leaving numerics wide is internally inconsistent.
• Codegen emits uncompilable code instead of failing loudly. valueForGeneralAssertion only emits the property when non-empty, but certificate/connection take a required, union-narrowed property. An empty property renders certificate().equals('x') (TS2554); an unknown one renders certificate('notAfter').equals('x') (TS2345). Either way checkly import succeeds silently and the imported project won't compile, with no diagnostic pointing at the offending assertion — contrast an unknown source, which throws cleanly via unsupportedAssertionSource. Worth mirroring that for properties.
• Export CertificateProperty / ConnectionProperty. traceroute-assertion.ts:10 exports its analogue and constructs/index.ts:58 re-exports the whole module. Users building typed wrappers can't currently name the type, and exporting is a prerequisite for the drift fix above.
• Breaking change has no migration path. SslMonitor shipped in v8.15.0/v8.16.0. With no CHANGELOG, an upgrader sees only Property 'certExpiresInDays' does not exist on type 'typeof SslAssertionBuilder' with no pointer to certificate('daysUntilExpiry'). Hard removal is defensible at two minors old with likely near-zero adoption — but it should be a conscious call, and the old→new mapping needs to reach users through release notes at minimum, since nothing in the repo carries it.
• Stale result-display description. Scoping out the result path is a reasonable line to draw, but formatters/assertion-line.ts:128 cites CERT_NOT_EXPIRED as its motivating example for a source this PR deletes. Cheap to reword now, archaeological in six months.

Pre-existing, not introduced here

Codegen for an out-of-set target (e.g. a future TLS1.4 from the backend) emits code that fails to typecheck, since validation deliberately permits any target string while the builder narrows it. Confirmed, but the old TlsVersionAssertionBuilder had the identical hole — flagging only so it isn't mistaken for a regression.

[sorccu]

Follow-up: blocker addressed in 789b7d1

Pushed a fix for the blocker from my review above — both scaffolding templates are now on the property-scoped grammar:

-      SslAssertionBuilder.certExpiresInDays().greaterThan(30),
-      SslAssertionBuilder.chainTrusted().equals(true),
-      SslAssertionBuilder.hostnameVerified().equals(true),
-      SslAssertionBuilder.tlsVersion().equals('TLS1.3'),
+      SslAssertionBuilder.certificate('daysUntilExpiry').greaterThan(30),
+      SslAssertionBuilder.connection('chainTrusted').equals(true),
+      SslAssertionBuilder.connection('hostnameVerified').equals(true),
+      SslAssertionBuilder.connection('tlsVersion').equals('TLS1.3'),

Applied to both examples/advanced-project/src/__checks__/uptime/ssl.check.ts and examples/advanced-project-js/src/__checks__/uptime/ssl.check.js.

Since nothing in CI exercises these files, I verified the port three ways rather than trusting the edit:

• Types — the four calls typecheck clean under tsc -p tsconfig.json when placed in a non-excluded location.
• Runtime validation — driven through the real SslMonitor.validate(): fatal: false, zero observations. Worth doing: my first probe reported 4 fatal errors, which turned out to be my own harness bug (wrong argument order into validateSslAssertion), not a problem with the port.
• Wire payload — synthesizes to the expected shape, semantics unchanged:

  [{"source":"CERTIFICATE","comparison":"GREATER_THAN","property":"daysUntilExpiry","target":"30","regex":null},
   {"source":"CONNECTION","comparison":"EQUALS","property":"chainTrusted","target":"true","regex":null},
   {"source":"CONNECTION","comparison":"EQUALS","property":"hostnameVerified","target":"true","regex":null},
   {"source":"CONNECTION","comparison":"EQUALS","property":"tlsVersion","target":"TLS1.3","regex":null}]

A sweep across examples/, both packages' src and e2e, and skills/ found no other call sites of the twelve removed methods, so this was the last one.

Still open from the review — I've left these for you, since they're judgement calls on your own design rather than mechanical fixes:

• Major: the type narrowing (compile-time) block asserts nothing (src/**/__tests__ is excluded from tsconfig.json, no test.typecheck, ban-ts-comment: 0).
• Major: certificateProperties/connectionProperties typed Record<string, PropertyRule> can drift from the builder unions silently.
• The minors, and the breaking-change migration note.

Also worth a separate look: the underlying gap that let this blocker exist. examples/ is outside the pnpm workspace and untouched by any workflow, so template/API drift is invisible to CI — a future rename of SslAssertionBuilder.certificate would break npm create checkly exactly the same way, with CI green throughout. The create-cli e2e already copies the local template via CHECKLY_E2E_LOCAL_TEMPLATE_ROOT but only asserts files exist; having it parse the scaffolded project would close the loop. Happy to open a separate PR for that if useful.