Skip to content

Commit cdf176f

Browse files
danielpaulusUbuntuclaude
authored
fix(cli): SSL/gRPC/traceroute — valid assertion builders, honest reporter, rich checks get [SIM-286][SIM-287][SIM-291] (#1391)
* feat(reporters): per-type TRACEROUTE/GRPC/SSL result rendering (P4-CLI-RESULTS) Render failure-debug diagnostics for the three uptime monitor types across every CLI result surface, mirroring the 4.1 public check-results fields: - rest/check-results.ts: typed TracerouteCheckResult/GrpcCheckResult/ SslCheckResult interfaces + additive fields on CheckResult/CheckResultField. - formatters/check-result-detail.ts: per-type terminal + markdown diagnostic block (checks results get) keyed on the typed fields. - reporters/util.ts: GRPC/SSL/TRACEROUTE branches in formatCheckResult (checkly test terminal), sourced from the runner artifact (checkRunData). - formatters/batch-stats.ts: add the three types to TIMING_TYPES. - reporters/json.ts: emit a per-type diagnostics object in the JSON report for a failed run (checkly test --reporter json). Snapshot + assertion tests for each type across all three surfaces. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(constructs): add GrpcMonitor, SslMonitor, TracerouteMonitor IaC constructs Implement the missing CLI constructs so users can author and deploy gRPC, SSL and Traceroute uptime monitors via checkly/constructs, and so backend CLI export templates that import these classes compile. Constructs follow the DNS/TCP/ICMP monitor pattern (extend Monitor, register with Session, validate, synthesize): - GrpcMonitor / GrpcRequest / GrpcConfig / GrpcMetadata + GrpcAssertionBuilder (RESPONSE_TIME, GRPC_STATUS_CODE, GRPC_HEALTHCHECK_STATUS, GRPC_RESPONSE, GRPC_METADATA). checkType GRPC, top-level degraded/maxResponseTime (<=30000). - SslMonitor / SslRequest / SslConfig / SecurityBaseline + SslAssertionBuilder (cert expiry, chain, hostname, TLS version, cipher, key size, etc). checkType SSL; response-time limits live in sslConfig (degraded/maxResponseTimeMs). - TracerouteMonitor / TracerouteRequest + TracerouteAssertionBuilder (RESPONSE_TIME, HOP_COUNT, PACKET_LOSS). checkType TRACEROUTE, top-level degraded/maxResponseTime (<=30000); port dropped for ICMP probes. Request shapes mirror the public API (verified against checkly-go-sdk types and the p5-parity captured payloads). Wires exports in constructs/index.ts, import codegen in check-codegen.ts (GRPC/SSL/TRACEROUTE -> *MonitorCodegen), and adds the three types to constants.CheckTypes. Tests: construct synthesize/validation/grouping specs, codegen specs (incl. ICMP port-strip and assertion builders), and a regression spec asserting the backend-style export snippets compile against checkly/constructs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(deploy): skip code-bundle upload when nothing was bundled archive.store() (the 'Uploading Playwright tests' step) was called unconditionally on every deploy. For a project of only uptime monitors (GRPC/SSL/TRACEROUTE), the bundler registers no files, so this uploaded an empty Playwright bundle — an unnecessary code-bundle upload in production, and a hard failure in devenv where it 500s on the storage backend. The remote code bundle is consumed only by Playwright check suites (via bundler.marker -> playwright-check.ts); browser checks upload snapshots separately. Add Bundler.isEmpty and skip store() when no files were registered. Refs T65. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cli): wire gRPC/SSL/Traceroute analytics paths + connection-error rendering Addresses review (Simo) new-monitor checklist gaps: - src/rest/analytics.ts: add checkTypeToPath (grpc-checks/ssl/traceroute) and defaultMetrics for the three types, matching the backend analytics routes + metric registries (gRPC total_*, SSL handshakeTimeMs_*/daysUntilExpiry_avg, TRACEROUTE finalHopLatencyAvg_*/totalHops_avg). - src/reporters/util.ts: add the Connection Error subsection to the GRPC/SSL/ TRACEROUTE result blocks, mirroring ICMP/DNS. * docs(cli): AI context references + examples for gRPC/SSL/Traceroute monitors Addresses review (Simo) new-monitor checklist Phases 5 & 7: - ai-context/references/configure-{grpc,ssl,traceroute}-monitors.md + REFERENCES and EXAMPLE_CONFIGS entries in context.ts (inline exampleConfig). - examples/advanced-project{,-js}/src/__checks__/uptime/{grpc,ssl,traceroute} TS + JS examples with group, assertions, and doc-link comments. * test(cli): e2e deploy fixtures + Create assertions for gRPC/SSL/Traceroute Addresses review (Simo) new-monitor checklist Phase 6: - e2e/__tests__/fixtures/deploy-project/{grpc,ssl,traceroute}.check.ts (activated:false) - deploy.spec.ts: assert the three logical IDs in the Create output. * feat(cli): support TEXT_BODY assertions for gRPC monitors Mirror monorepo #2730 which added TEXT_BODY to grpcMonitorAssertionSources. Adds GrpcAssertionBuilder.textBody() (source TEXT_BODY) + codegen case, matching the API check's textBody() builder. * refactor(cli): restructure SslMonitor props per review (wire shape unchanged) Per @sorccu's review: make SslMonitor ergonomics consistent with other monitors. - degradedResponseTime/maxResponseTime -> construct top level (were in sslConfig) - hostname/port/ipFamily -> request top level (were in sslConfig) - sslClientCertificateId -> into sslConfig (was on request) synthesize() remaps to the identical API wire shape (request.sslConfig.* + request.sslClientCertificateId), so the payload the backend receives is unchanged. Updated codegen (wire->construct), unit + codegen + export-snippet tests, examples (ts+js), e2e fixture, and AI context. * refactor(cli): drop Ms suffix on SslConfig.handshakeTimeout per review @sorccu: we don't suffix timeout properties with Ms even when the value is milliseconds. Renames the construct field handshakeTimeoutMs -> handshakeTimeout; synthesize() and the codegen wire interface keep the API's handshakeTimeoutMs. * refactor(cli): emit SSL degraded/maxResponseTime at top level to match gRPC/traceroute wire * fix(reporters): don't render green assertions on a request error; add gRPC/SSL detail [SIM-286] Suppress the Assertions block when a result has a requestError (nothing was evaluated); humanize gRPC assertion source labels; render gRPC response time; add an SSL response-time reason line and a security-baseline verdict line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * fix(constructs): make SSL/traceroute assertion builders emit backend-valid payloads [SIM-287] responseTime() defaults property to 'avg' + avg/min/max/stdDev selectors; SslBaselineSeverity 'warn' -> 'degrade'; add greaterThanOrEqual and matches builders; add SSL sources OCSP_STAPLED/HANDSHAKE_TIME_MS/SAN_CONTAINS; add degraded<=max validation. Codegen updated to round-trip all of the above. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * feat(cli): show assertions + rich per-type detail in checkly checks get [SIM-291] checkly checks get --result rendered the per-type RESULT block but omitted the assertions (so a failing SSL/gRPC/traceroute check showed no reason) and a lot of available detail. Bring it on par with checkly test and surface the rich result body agents can use: - Extract the assertion-line renderer into a shared formatters/assertion-line.ts used by BOTH the checkly test reporter and checks get, so they render identically and cannot drift. - Add an ASSERTIONS section to the SSL/gRPC/traceroute detail formatters (terminal + markdown), including a Received value for falsy actuals (0/false/''). - gRPC: add a TIMING breakdown (dns/connect/total). - SSL: add a CERTIFICATE section (subject/issuer CN, validity, key, signature, SHA-256 fingerprint, SANs, serial, self-signed/CA, OCSP stapled) and a per-rule SECURITY BASELINE breakdown, keeping the one-line baseline summary. - Traceroute: add probe protocol + DNS timing. - Harden the shared truncate() to cap on the stringified length (objects/numbers no longer bypass it) and slice by code point (no split surrogate pairs); give the markdown request-error its own heading. The --output json path already emits the full result body and is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * test(cli): cover scalar baseline rule + self-signed/CA cert flags [SIM-291] Adds the two branches the tri-model review flagged as untested: a scalar securityBaseline rule (rendered as key: value) and selfSigned/isCA=true cert flags. Test-only; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * fix(grpc): raise response-time cap to 180000ms in the CLI construct [SIM-287] GrpcMonitor capped degradedResponseTime/maxResponseTime at 30000 (JSDoc @maximum and the validate() bounds), but gRPC calls run to the 180s timeout and the backend allows up to 180000 (grpcResponseTimeLimitFields). Users couldn't set a gRPC threshold above 30s. Raise the cap to 180000 to match the backend (and the terraform provider fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * style(cli): single-quote strings in ssl-assertion-codegen spec [SIM-287] Fixes 4 @stylistic/quotes lint errors (strings must use single quotes) in ssl-assertion-codegen.spec.ts — the affected `expected` strings contain inner single quotes and were written with double quotes; the repo eslint config requires single quotes with escaping. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * fix(cli): stabilize assertion symbols across platforms * refactor(cli): drop greaterThanOrEqual, matches, handshakeTime, sanContains from SSL assertions [SIM-287] Mirror the removals made on #1392 so the two SSL-assertion PRs stay consistent and don't resurrect this API when they merge: - Remove greaterThanOrEqual (GREATER_THAN_OR_EQUAL) and matches (MATCHES) from the shared Numeric/General assertion builders + the Comparison union, and their codegen cases in internal/assertion-codegen.ts. - Remove the handshakeTimeMs()/HANDSHAKE_TIME_MS and sanContains()/SAN_CONTAINS SSL sources (builder + union + ssl-assertion-codegen cases). - Update ssl-monitor + ssl-assertion-codegen specs to the remaining backend-valid operators (equals/notEquals). Keeps keySizeBits/certExpiresInDays. The SSL result's handshakeTimeMs response field is unaffected. Backend still supports all of these operators/sources, so they can be re-exposed later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc * style(cli): remove padded blank line in ssl-assertion-codegen spec [SIM-287] Fixes @stylistic/padded-blocks after dropping the removed-operators test block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NC9pESYE5wGqy5TKRzufyc --------- Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-108.eu-central-1.compute.internal> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent ee3eab5 commit cdf176f

37 files changed

Lines changed: 2409 additions & 314 deletions

packages/cli/src/ai-context/references/configure-traceroute-monitors.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
- Use `request.maxHops` (default: 30) and `request.maxUnknownHops` (default: 15) to control trace depth.
99
- Use `degradedResponseTime` and `maxResponseTime` (milliseconds) to configure response time thresholds.
1010
- Traceroute assertions support `RESPONSE_TIME`, `HOP_COUNT`, and `PACKET_LOSS` sources.
11+
- `TracerouteAssertionBuilder.responseTime()` takes an optional property — `'avg'` (default), `'min'`, `'max'`, or `'stdDev'`. `hopCount()` and `packetLoss()` take no property.
1112
- **Plan-gated properties:** `retryStrategy`, `runParallel`, and higher frequencies are not available on all plans. Check entitlements matching `UPTIME_CHECKS_*` before using these. Omit any property whose entitlement is disabled. See `npx checkly skills manage` for details.
1213

1314
<!-- EXAMPLE: TRACEROUTE_MONITOR -->

packages/cli/src/constructs/__tests__/grpc-monitor.spec.ts

Lines changed: 119 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -109,19 +109,49 @@ describe('GrpcMonitor', () => {
109109
})
110110

111111
describe('validation', () => {
112-
it('should error if degradedResponseTime is above 30000', async () => {
112+
it('should error if degradedResponseTime is above 180000', async () => {
113113
setupProject()
114114
const check = new GrpcMonitor('test-check', {
115115
name: 'Test Check',
116116
request,
117-
degradedResponseTime: 30001,
117+
degradedResponseTime: 180001,
118118
})
119119
const diags = new Diagnostics()
120120
await check.validate(diags)
121121
expect(diags.isFatal()).toEqual(true)
122122
expect(diags.observations).toEqual(expect.arrayContaining([
123123
expect.objectContaining({
124-
message: expect.stringContaining('The value of "degradedResponseTime" must be 30000 or lower.'),
124+
message: expect.stringContaining('The value of "degradedResponseTime" must be 180000 or lower.'),
125+
}),
126+
]))
127+
})
128+
129+
it('should not error for a response time above 30000 (gRPC allows up to 180000)', async () => {
130+
setupProject()
131+
const check = new GrpcMonitor('test-check', {
132+
name: 'Test Check',
133+
request,
134+
degradedResponseTime: 90000,
135+
maxResponseTime: 120000,
136+
})
137+
const diags = new Diagnostics()
138+
await check.validate(diags)
139+
expect(diags.isFatal()).toEqual(false)
140+
})
141+
142+
it('should error when degradedResponseTime exceeds the default maxResponseTime', async () => {
143+
setupProject()
144+
const check = new GrpcMonitor('test-check', {
145+
name: 'Test Check',
146+
request,
147+
degradedResponseTime: 30000,
148+
})
149+
const diags = new Diagnostics()
150+
await check.validate(diags)
151+
expect(diags.isFatal()).toEqual(true)
152+
expect(diags.observations).toEqual(expect.arrayContaining([
153+
expect.objectContaining({
154+
message: expect.stringContaining('must be less than or equal to the default "maxResponseTime" of 20000'),
125155
}),
126156
]))
127157
})
@@ -138,5 +168,91 @@ describe('GrpcMonitor', () => {
138168
await check.validate(diags)
139169
expect(diags.isFatal()).toEqual(false)
140170
})
171+
172+
it('should error on an assertion with an unknown source', async () => {
173+
setupProject()
174+
const check = new GrpcMonitor('test-check', {
175+
name: 'Test Check',
176+
request: {
177+
...request,
178+
// HOP_COUNT belongs to traceroute, not gRPC.
179+
assertions: [{ source: 'HOP_COUNT', property: '', comparison: 'LESS_THAN', target: '5', regex: null }],
180+
},
181+
})
182+
const diags = new Diagnostics()
183+
await check.validate(diags)
184+
expect(diags.isFatal()).toEqual(true)
185+
expect(diags.observations).toEqual(expect.arrayContaining([
186+
expect.objectContaining({
187+
message: expect.stringContaining(
188+
'The assertion at "request.assertions[0]" has an unknown source "HOP_COUNT".',
189+
),
190+
}),
191+
]))
192+
})
193+
194+
it('should error on an assertion with an unsupported comparison', async () => {
195+
setupProject()
196+
const check = new GrpcMonitor('test-check', {
197+
name: 'Test Check',
198+
request: {
199+
...request,
200+
// GREATER_THAN_OR_EQUAL is an SSL-only operator, not in the gRPC set.
201+
assertions: [
202+
{ source: 'GRPC_STATUS_CODE', property: '', comparison: 'GREATER_THAN_OR_EQUAL', target: '0', regex: null },
203+
],
204+
},
205+
})
206+
const diags = new Diagnostics()
207+
await check.validate(diags)
208+
expect(diags.isFatal()).toEqual(true)
209+
expect(diags.observations).toEqual(expect.arrayContaining([
210+
expect.objectContaining({
211+
message: expect.stringContaining('has an unsupported comparison "GREATER_THAN_OR_EQUAL"'),
212+
}),
213+
]))
214+
})
215+
216+
it('should accept every comparison in the backend gRPC set', async () => {
217+
setupProject()
218+
// Pins the full accepted set so silently dropping one from the validator's list
219+
// is caught, since it is not compile-checked against a union.
220+
const comparisons = [
221+
'EQUALS', 'NOT_EQUALS', 'HAS_KEY', 'NOT_HAS_KEY', 'HAS_VALUE', 'NOT_HAS_VALUE',
222+
'IS_EMPTY', 'NOT_EMPTY', 'GREATER_THAN', 'LESS_THAN', 'CONTAINS', 'NOT_CONTAINS',
223+
'IS_NULL', 'NOT_NULL',
224+
]
225+
const check = new GrpcMonitor('test-check', {
226+
name: 'Test Check',
227+
request: {
228+
...request,
229+
assertions: comparisons.map(comparison => (
230+
{ source: 'GRPC_RESPONSE', property: '', comparison, target: 'x', regex: null }
231+
)),
232+
},
233+
})
234+
const diags = new Diagnostics()
235+
await check.validate(diags)
236+
expect(diags.isFatal()).toEqual(false)
237+
})
238+
239+
it('should not error on a property carried by any gRPC source', async () => {
240+
setupProject()
241+
const check = new GrpcMonitor('test-check', {
242+
name: 'Test Check',
243+
request: {
244+
...request,
245+
// gRPC places no source/property constraint, so a property on RESPONSE_TIME
246+
// (which the traceroute validator would reject) is accepted here.
247+
assertions: [
248+
{ source: 'RESPONSE_TIME', property: 'avg', comparison: 'LESS_THAN', target: '5000', regex: null },
249+
GrpcAssertionBuilder.responseMetadata('content-type').contains('grpc'),
250+
],
251+
},
252+
})
253+
const diags = new Diagnostics()
254+
await check.validate(diags)
255+
expect(diags.isFatal()).toEqual(false)
256+
})
141257
})
142258
})
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
import { describe, it, expect } from 'vitest'
2+
3+
import { valueForSslAssertion } from '../ssl-assertion-codegen.js'
4+
import { SslAssertion } from '../ssl-assertion.js'
5+
import { GeneratedFile, Output } from '../../sourcegen/index.js'
6+
7+
function render (assertion: SslAssertion): string {
8+
const output = new Output()
9+
const file = new GeneratedFile('foo.ts')
10+
const result = valueForSslAssertion(file, assertion)
11+
result.render(output)
12+
return output.finalize()
13+
}
14+
15+
describe('SSL Assertion Codegen', () => {
16+
it('generates the new SSL assertion sources', () => {
17+
const cases: { input: SslAssertion, expected: string }[] = [
18+
// Boolean sources emit a boolean literal, not a quoted string.
19+
{
20+
input: { source: 'OCSP_STAPLED', property: '', comparison: 'EQUALS', target: 'true', regex: null },
21+
expected: 'SslAssertionBuilder.ocspStapled().equals(true)\n',
22+
},
23+
{
24+
input: { source: 'CERT_NOT_EXPIRED', property: '', comparison: 'EQUALS', target: 'true', regex: null },
25+
expected: 'SslAssertionBuilder.certNotExpired().equals(true)\n',
26+
},
27+
{
28+
input: { source: 'CHAIN_TRUSTED', property: '', comparison: 'EQUALS', target: 'false', regex: null },
29+
expected: 'SslAssertionBuilder.chainTrusted().equals(false)\n',
30+
},
31+
// Numeric key size emits an unquoted number.
32+
{
33+
input: { source: 'KEY_SIZE_BITS', property: '', comparison: 'EQUALS', target: '2048', regex: null },
34+
expected: 'SslAssertionBuilder.keySizeBits().equals(2048)\n',
35+
},
36+
// The string sources support the MATCHES (regex) operator.
37+
{
38+
input: { source: 'CIPHER_SUITE', property: '', comparison: 'MATCHES', target: 'TLS_(AES|CHACHA)', regex: null },
39+
expected: 'SslAssertionBuilder.cipherSuite().matches(\'TLS_(AES|CHACHA)\')\n',
40+
},
41+
]
42+
for (const test of cases) {
43+
expect(render(test.input)).toEqual(test.expected)
44+
}
45+
})
46+
})

packages/cli/src/constructs/__tests__/ssl-assertion.spec.ts

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,4 +98,46 @@ describe('SslAssertionBuilder — typed target values', () => {
9898
expect(SslAssertionBuilder.cipherSuite().equals('TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256')).toBeTruthy()
9999
})
100100
})
101+
102+
describe('matches() — regex operator on the string sources', () => {
103+
it('produces a MATCHES assertion for cipherSuite / issuerCn / signatureAlgorithm', () => {
104+
expect(SslAssertionBuilder.cipherSuite().matches('TLS_(AES|CHACHA)')).toMatchObject({
105+
source: 'CIPHER_SUITE', comparison: 'MATCHES', target: 'TLS_(AES|CHACHA)',
106+
})
107+
expect(SslAssertionBuilder.issuerCn().matches('^Let\'s Encrypt')).toMatchObject({
108+
source: 'ISSUER_CN', comparison: 'MATCHES', target: '^Let\'s Encrypt',
109+
})
110+
expect(SslAssertionBuilder.signatureAlgorithm().matches('SHA(256|384)')).toMatchObject({
111+
source: 'SIGNATURE_ALGORITHM', comparison: 'MATCHES', target: 'SHA(256|384)',
112+
})
113+
})
114+
})
115+
116+
describe('per-source builders reject operators the backend does not allow', () => {
117+
it('rejects them at compile time', () => {
118+
// Compile-time-only checks: the removed methods do not exist at runtime, so the
119+
// body is type-checked but never executed (an uncalled function).
120+
121+
const _typeChecks = () => {
122+
// keySizeBits supports EQUALS only (GREATER_THAN_OR_EQUAL was dropped)
123+
// @ts-expect-error greaterThan is not available on the key-size builder
124+
SslAssertionBuilder.keySizeBits().greaterThan(2048)
125+
// @ts-expect-error notEquals is not available on the key-size builder
126+
SslAssertionBuilder.keySizeBits().notEquals(2048)
127+
// boolean sources take a boolean, not the string 'true'
128+
// @ts-expect-error 'true' (string) is not assignable to boolean
129+
SslAssertionBuilder.certNotExpired().equals('true')
130+
// matches is not offered on non-string sources
131+
// @ts-expect-error matches is not available on the cert-not-expired builder
132+
SslAssertionBuilder.certNotExpired().matches('x')
133+
// the general string operators are not offered on SSL string sources
134+
// @ts-expect-error contains is not available on the cipher-suite builder
135+
SslAssertionBuilder.cipherSuite().contains('x')
136+
// tlsVersion supports EQUALS only
137+
// @ts-expect-error notEquals is not available on the tls-version builder
138+
SslAssertionBuilder.tlsVersion().notEquals('TLS1.3')
139+
}
140+
expect(_typeChecks).toBeDefined()
141+
})
142+
})
101143
})

0 commit comments

Comments
 (0)