Skip to content

Commit 4a19753

Browse files
committed
feat: add cloud-provider-regression skill and increase benchmark timeout
- Increase local /metrics scrape timeout to 5 minutes to accommodate server-side metric compilation for very large scale zero-copy graphs. - Add new cloud-provider-regression analysis skill.
1 parent 07d3f01 commit 4a19753

File tree

4 files changed

+582
-1
lines changed

4 files changed

+582
-1
lines changed
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
name: Cloud Provider Regression Test
3+
description: Connectivity, chat, JSON & streaming regression tests for all enabled cloud LLM providers
4+
version: 1.0.0
5+
category: analysis
6+
runtime: node
7+
entry: scripts/run-regression.cjs
8+
install: npm
9+
requirements:
10+
node: ">=18"
11+
npm_install: false
12+
platforms: ["linux", "macos", "windows"]
13+
---
14+
15+
# Cloud Provider Regression Test
16+
17+
Tests every enabled cloud provider for connectivity, chat completion, JSON output, and SSE streaming.
18+
Reads keys from `~/.aegis-ai/llm-config.json`.
19+
20+
## Standalone
21+
22+
```bash
23+
node scripts/run-regression.cjs # all providers
24+
node scripts/run-regression.cjs --provider glm,xai # specific
25+
node scripts/run-regression.cjs --verbose # full responses
26+
```
27+
28+
## Protocol
29+
30+
```jsonl
31+
{"event":"ready","providers":8}
32+
{"event":"test_result","suite":"GLM","test":"chat","status":"pass","timeMs":1930}
33+
{"event":"complete","passed":14,"failed":1,"total":15,"timeMs":38000}
34+
```
35+
36+
## Tests Per Provider
37+
38+
| Test | Verifies |
39+
|------|----------|
40+
| Chat | Connectivity, auth, URL construction, param compat |
41+
| JSON | Structured output (JSON instruction following) |
42+
| Stream | SSE streaming, chunks received |
43+
44+
Results saved to `~/.aegis-ai/regression-tests/`.
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
params:
2+
- key: providers
3+
label: Provider Filter
4+
type: string
5+
default: "all"
6+
description: "Comma-separated provider IDs to test, or 'all'"
7+
8+
- key: skipJson
9+
label: Skip JSON test
10+
type: boolean
11+
default: false
12+
description: Skip the JSON output test
13+
14+
- key: skipStream
15+
label: Skip streaming test
16+
type: boolean
17+
default: false
18+
description: Skip the SSE streaming test
19+
20+
- key: timeout
21+
label: Request Timeout (ms)
22+
type: number
23+
default: 30000
24+
description: Per-request timeout in milliseconds

0 commit comments

Comments
 (0)