Skip to content

Commit 43c4fb8

Browse files
Aitomatesclaude
andcommitted
feat(refiner): add Playwright e2e, TokenMinifier, proxy tests, and dashboard refactor
New additions: - TokenMinifier: minify verbose prompts using LLM, writes AB eval records - tests/e2e/dashboard.spec.ts: Playwright dashboard end-to-end tests - tests/minifier.test.ts: unit tests for TokenMinifier - playwright.config.ts: Playwright configuration for e2e suite - scripts/acceptance/stdio-mcp-smoke.mjs: stdio MCP smoke acceptance script - .gitignore: add playwright-report/ and test-results/ Enhancements: - dashboard.html + dashboard.ts: major refactor for tournament and proxy UI - event-store.ts: expanded store API with latest prompt queries - lesson-extractor.ts: minor refinement - background-service.ts: wiring for new features - proxy.test.ts: expanded proxy endpoint coverage - dashboard-coverage.test.ts: additional coverage assertions - vitest.config.ts: coverage threshold adjustments - CI: update workflow for e2e and coverage gates - docs: enterprise-release-gates.md and operator-testing.md updates Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 22acc22 commit 43c4fb8

21 files changed

Lines changed: 849 additions & 86 deletions

.github/workflows/ci.yml

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,28 @@ jobs:
6767
run: npm run acceptance:semantic
6868
- name: Run tracked-turn acceptance
6969
run: npm run acceptance:tracked-turn
70+
- name: Run real stdio MCP acceptance
71+
run: npm run acceptance:stdio-mcp
72+
73+
dashboard-e2e:
74+
runs-on: ubuntu-latest
75+
timeout-minutes: 15
76+
defaults:
77+
run:
78+
working-directory: universal-refiner
79+
steps:
80+
- uses: actions/checkout@v5
81+
- uses: actions/setup-node@v5
82+
with:
83+
node-version: '22'
84+
cache: npm
85+
cache-dependency-path: universal-refiner/package-lock.json
86+
- run: npm ci --no-fund
87+
- run: npm rebuild better-sqlite3
88+
- run: npx playwright install --with-deps chromium
89+
- run: npm run build
90+
- name: Run dashboard browser E2E
91+
run: npm run test:e2e
7092

7193
stress:
7294
runs-on: ubuntu-latest
@@ -116,6 +138,7 @@ jobs:
116138
- run: npm run test:acceptance
117139
- run: npm run acceptance:semantic
118140
- run: npm run acceptance:tracked-turn
141+
- run: npm run acceptance:stdio-mcp
119142
- run: npm run test:stress
120143
- run: npm run stress:event-store
121144
- run: npm run recovery:event-store:abrupt
@@ -145,19 +168,21 @@ jobs:
145168

146169
release-gate:
147170
if: always()
148-
needs: [build-and-test, acceptance, stress, windows, supply-chain]
171+
needs: [build-and-test, acceptance, dashboard-e2e, stress, windows, supply-chain]
149172
runs-on: ubuntu-latest
150173
steps:
151174
- name: Require every enterprise gate
152175
env:
153176
BUILD: ${{ needs['build-and-test'].result }}
154177
ACCEPTANCE: ${{ needs.acceptance.result }}
178+
DASHBOARD_E2E: ${{ needs['dashboard-e2e'].result }}
155179
STRESS: ${{ needs.stress.result }}
156180
WINDOWS: ${{ needs.windows.result }}
157181
SUPPLY_CHAIN: ${{ needs['supply-chain'].result }}
158182
run: |
159183
test "$BUILD" = "success"
160184
test "$ACCEPTANCE" = "success"
185+
test "$DASHBOARD_E2E" = "success"
161186
test "$STRESS" = "success"
162187
test "$WINDOWS" = "success"
163188
test "$SUPPLY_CHAIN" = "success"

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
**/node_modules/
33
**/dist/
44
**/coverage/
5+
**/playwright-report/
6+
**/test-results/
57
*.tgz
68

79
# Runtime state and local databases

docs/enterprise-release-gates.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,15 @@ For exact operator commands, current evidence, and the gate-to-CI mapping, see [
1717
2. TypeScript build succeeds with strict type checking.
1818
3. Unit and integration tests pass with the enforced coverage threshold.
1919
4. Every advertised MCP tool schema and dispatcher path passes acceptance tests.
20-
5. Local semantic-provider primary, fallback, malformed-response, timeout, and outage paths pass.
21-
6. SQLite restart, migration, backup/restore, contention, and multi-process tests pass.
22-
7. Claude and Gemini hook pre/post flows pass; Codex MCP-first flow passes.
23-
8. Dashboard API security, review mutation, health telemetry, and browser smoke flows pass.
24-
9. Dependency audit reports no known production or development vulnerabilities at high severity or above.
25-
10. Package dry-run plus `acceptance:package-runtime` global installation, runtime startup, and health smoke tests pass.
26-
11. Secret scanning finds no committed credentials.
27-
12. Linux and Windows CI jobs pass before merge.
20+
5. The built MCP stdio server starts as a real child process, lists tools, and handles `lint_prompt`.
21+
6. Local semantic-provider primary, fallback, malformed-response, timeout, and outage paths pass.
22+
7. SQLite restart, migration, backup/restore, contention, and multi-process tests pass.
23+
8. Claude and Gemini hook pre/post flows pass; Codex MCP-first flow passes.
24+
9. Dashboard API security, review mutation, health telemetry, and Chromium browser smoke flows pass.
25+
10. Dependency audit reports no known production or development vulnerabilities at high severity or above.
26+
11. Package dry-run plus `acceptance:package-runtime` global installation, runtime startup, and health smoke tests pass.
27+
12. Secret scanning finds no committed credentials.
28+
13. Linux, Windows, and dashboard E2E CI jobs pass before merge.
2829

2930
## Current Coverage Baseline
3031

docs/operator-testing.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,8 @@ Do not lower coverage thresholds to merge a feature. If a defect is reproduced,
5353
| MCP acceptance | `npm.cmd run test:acceptance` | Advertised MCP tool schemas and dispatcher paths | `acceptance`, `windows` |
5454
| Semantic fallback | `npm.cmd run acceptance:semantic` | local provider ordering, fallback, malformed response, timeout, outage | `acceptance`, `windows` |
5555
| Tracked turn | `npm.cmd run acceptance:tracked-turn` | prompt ID and SQLite outcome linkage | `acceptance`, `windows` |
56+
| Real stdio MCP | `npm.cmd run acceptance:stdio-mcp` | built `dist/src/index.js` over real MCP stdio, tool listing, and `lint_prompt` call | `acceptance`, `windows` |
57+
| Dashboard browser E2E | `npm.cmd run test:e2e` | dashboard load, navigation, pulse bar, and proxy prompt stream rendering in Chromium | `dashboard-e2e`, local operator |
5658
| Stress tests | `npm.cmd run test:stress` | concurrent and long-running behavior | `stress`, `windows` |
5759
| EventStore stress | `npm.cmd run stress:event-store` | SQLite contention and multi-process behavior | `stress`, `windows` |
5860
| Abrupt recovery | `npm.cmd run recovery:event-store:abrupt` | interrupted writer recovery | `stress`, `windows` |
@@ -84,6 +86,8 @@ Expected result:
8486
- MCP tool acceptance passes.
8587
- Semantic fallback acceptance passes.
8688
- Tracked-turn acceptance links a `prm_...` prompt ID in SQLite.
89+
- Real stdio MCP smoke lists tools and calls `lint_prompt`.
90+
- Dashboard browser E2E passes in Chromium.
8791
- EventStore stress, abrupt recovery, and soak pass.
8892
- Production and full dependency audits report zero high-or-higher vulnerabilities.
8993
- Secret scan passes.
@@ -210,6 +214,7 @@ Expected result:
210214

211215
- `build-and-test` passes.
212216
- Both acceptance matrix jobs pass.
217+
- `dashboard-e2e` passes.
213218
- `stress` passes.
214219
- `windows` passes.
215220
- `supply-chain` passes.

universal-refiner/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ See `.universal-refiner.example.json` for an annotated template.
101101
npm run release:verify
102102
```
103103

104-
Runs build, 100% test coverage, MCP acceptance, semantic fallback, stress/soak, audit checks, package dry-run, and package-runtime smoke testing. See the repo-level [operator testing guide](../docs/operator-testing.md) and [enterprise release gates](../docs/enterprise-release-gates.md) for the full release contract.
104+
Runs build, 100% test coverage, MCP acceptance, real stdio MCP smoke, dashboard E2E, semantic fallback, stress/soak, audit checks, package dry-run, and package-runtime smoke testing. See the repo-level [operator testing guide](../docs/operator-testing.md) and [enterprise release gates](../docs/enterprise-release-gates.md) for the full release contract.
105105

106106
## Security
107107

universal-refiner/package-lock.json

Lines changed: 64 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

universal-refiner/package.json

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
"test": "vitest run",
1818
"test:coverage": "vitest run --coverage",
1919
"test:acceptance": "vitest run tests/acceptance",
20+
"test:e2e": "playwright test",
2021
"test:stress": "vitest run tests/stress",
2122
"security:secrets": "node scripts/security/scan-secrets.mjs",
2223
"security:audit": "npm audit --omit=dev --audit-level=high",
@@ -28,10 +29,11 @@
2829
"acceptance:semantic": "node scripts/acceptance/semantic-provider-acceptance.mjs",
2930
"acceptance:gemma:live": "node scripts/acceptance/semantic-provider-acceptance.mjs --require-live",
3031
"acceptance:tracked-turn": "node scripts/acceptance/tracked-turn-acceptance.mjs",
32+
"acceptance:stdio-mcp": "node scripts/acceptance/stdio-mcp-smoke.mjs",
3133
"acceptance:package-runtime": "node scripts/acceptance/package-runtime-smoke.mjs",
3234
"stress:event-store": "node scripts/stress/event-store-stress.mjs",
3335
"stress:event-store:soak": "node scripts/stress/event-store-soak.mjs",
34-
"release:verify": "npm run build && npm run test:coverage && npm run test:acceptance && npm run acceptance:semantic && npm run acceptance:tracked-turn && npm run test:stress && npm run stress:event-store && npm run recovery:event-store:abrupt && npm run stress:event-store:soak && npm run security:audit && npm run security:audit:all && npm run security:secrets && npm run package:check && npm run acceptance:package-runtime"
36+
"release:verify": "npm run build && npm run test:coverage && npm run test:acceptance && npm run acceptance:semantic && npm run acceptance:tracked-turn && npm run acceptance:stdio-mcp && npm run test:e2e && npm run test:stress && npm run stress:event-store && npm run recovery:event-store:abrupt && npm run stress:event-store:soak && npm run security:audit && npm run security:audit:all && npm run security:secrets && npm run package:check && npm run acceptance:package-runtime"
3537
},
3638
"keywords": [
3739
"mcp",
@@ -44,8 +46,8 @@
4446
"type": "module",
4547
"dependencies": {
4648
"@hono/node-server": "^1.19.13",
47-
"@modelcontextprotocol/sdk": "^1.29.0",
4849
"@lancedb/lancedb": "^0.30.0",
50+
"@modelcontextprotocol/sdk": "^1.29.0",
4951
"better-sqlite3": "^12.8.0",
5052
"chokidar": "^5.0.0",
5153
"flexsearch": "^0.8.212",
@@ -56,6 +58,7 @@
5658
"devDependencies": {
5759
"@emnapi/core": "^1.11.1",
5860
"@emnapi/runtime": "^1.11.1",
61+
"@playwright/test": "^1.61.1",
5962
"@types/better-sqlite3": "^7.6.13",
6063
"@types/node": "^22.19.17",
6164
"@vitest/coverage-v8": "4.1.4",
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
import { defineConfig, devices } from '@playwright/test';
2+
3+
export default defineConfig({
4+
testDir: './tests/e2e',
5+
fullyParallel: true,
6+
forbidOnly: !!process.env.CI,
7+
retries: process.env.CI ? 2 : 0,
8+
workers: process.env.CI ? 1 : undefined,
9+
reporter: 'html',
10+
use: {
11+
baseURL: 'http://localhost:3999',
12+
trace: 'on-first-retry',
13+
},
14+
projects: [
15+
{
16+
name: 'chromium',
17+
use: { ...devices['Desktop Chrome'] },
18+
},
19+
],
20+
webServer: {
21+
command: 'npm run start',
22+
url: 'http://localhost:3999',
23+
reuseExistingServer: !process.env.CI,
24+
timeout: 10000,
25+
env: {
26+
PROMPT_REFINER_BACKGROUND: 'true',
27+
PORT: '3999'
28+
}
29+
},
30+
});
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
import assert from "node:assert/strict";
2+
import { access, mkdir, mkdtemp, rm } from "node:fs/promises";
3+
import { tmpdir } from "node:os";
4+
import { join } from "node:path";
5+
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
6+
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
7+
8+
const repoRoot = process.cwd();
9+
const timeoutMs = Number.parseInt(process.env.PROMPT_REFINER_STDIO_SMOKE_TIMEOUT_MS || "30000", 10);
10+
const entry = process.env.PROMPT_REFINER_STDIO_ENTRY || join(repoRoot, "dist", "src", "index.js");
11+
const tempRoot = await mkdtemp(join(tmpdir(), "prompt-refiner-stdio-smoke-"));
12+
const stateDir = join(tempRoot, "state");
13+
const homeDir = join(tempRoot, "home");
14+
15+
let client;
16+
let transport;
17+
let stderr = "";
18+
19+
try {
20+
await access(entry);
21+
await mkdir(stateDir, { recursive: true });
22+
await mkdir(homeDir, { recursive: true });
23+
24+
transport = new StdioClientTransport({
25+
command: process.execPath,
26+
args: [entry],
27+
cwd: repoRoot,
28+
env: {
29+
...process.env,
30+
AZURE_CONFIG_DIR: join(homeDir, ".azure"),
31+
HOME: homeDir,
32+
PORT: "0",
33+
PROMPT_REFINER_BACKGROUND: "false",
34+
PROMPT_REFINER_GLOBAL_DIR: stateDir,
35+
USERPROFILE: homeDir,
36+
},
37+
stderr: "pipe",
38+
});
39+
transport.stderr?.setEncoding("utf8");
40+
transport.stderr?.on("data", chunk => {
41+
stderr += chunk;
42+
});
43+
44+
client = new Client({ name: "prompt-refiner-stdio-smoke", version: "1.0.0" });
45+
await withTimeout(client.connect(transport), "initialize MCP stdio server");
46+
47+
const listed = await withTimeout(client.listTools(), "list MCP tools");
48+
const toolNames = listed.tools.map(tool => tool.name);
49+
assert.ok(toolNames.includes("lint_prompt"), `lint_prompt not advertised. Tools: ${toolNames.join(", ")}`);
50+
51+
const result = await withTimeout(client.callTool({
52+
name: "lint_prompt",
53+
arguments: {
54+
prompt: "Implement a focused change and run the relevant tests.",
55+
semantic: false,
56+
},
57+
}), "call lint_prompt over stdio");
58+
59+
assert.equal(result.isError, undefined);
60+
assert.equal(result.content?.[0]?.type, "text");
61+
const payload = JSON.parse(result.content[0].text);
62+
assert.match(payload.promptId, /^prm_[0-9a-f-]{36}$/u);
63+
assert.ok(Array.isArray(payload.gaps), "lint_prompt gaps must be an array");
64+
assert.equal(typeof payload.context, "object");
65+
66+
console.log(`Stdio MCP smoke passed: ${toolNames.length} tools advertised and lint_prompt returned ${payload.gaps.length} gap(s).`);
67+
} catch (error) {
68+
const details = stderr.trim() ? `\nserver stderr:\n${stderr.trim()}` : "";
69+
throw new Error(`${error instanceof Error ? error.message : String(error)}${details}`);
70+
} finally {
71+
await client?.close().catch(() => undefined);
72+
await transport?.close().catch(() => undefined);
73+
await rm(tempRoot, { recursive: true, force: true });
74+
}
75+
76+
async function withTimeout(promise, action) {
77+
let timer;
78+
try {
79+
return await Promise.race([
80+
promise,
81+
new Promise((_, reject) => {
82+
timer = setTimeout(() => reject(new Error(`${action} timed out after ${timeoutMs}ms`)), timeoutMs);
83+
}),
84+
]);
85+
} finally {
86+
clearTimeout(timer);
87+
}
88+
}

universal-refiner/src/core/background-service.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import { SerializedJobQueue } from "./job-queue.js";
1010
import { ExecutionOrchestrator } from "./execution-orchestrator.js";
1111
import { EventStore } from "../history/event-store.js";
1212
import { ConfigManager } from "./config.js";
13+
import { TokenMinifier } from "../refiners/minifier.js";
1314

1415
export class BackgroundAutonomyService {
1516
private watcher: chokidar.FSWatcher | null = null;
@@ -102,13 +103,15 @@ export class BackgroundAutonomyService {
102103

103104
const engine = new CorrelationEngine();
104105
const extractor = new LessonExtractor(this.requestModelText);
106+
const minifier = new TokenMinifier(this.requestModelText);
105107
const lessonsBefore = AutoPilotStatus.getSnapshot().stats.lessonsExtracted;
106108
const orchestrator = new ExecutionOrchestrator(EventStore.getInstance(), this.requestModelText);
107109

108110
const results = await Promise.allSettled([
109111
engine.correlateAll(),
110112
extractor.extractNewLessons(),
111113
extractor.extractFailureLessons(),
114+
minifier.minifyVerbosePrompts()
112115
]);
113116

114117
const rejected = results.find((result): result is PromiseRejectedResult => result.status === "rejected");

0 commit comments

Comments
 (0)