Skip to content

Commit 23ffb65

Browse files
data-douserCopilotCopilot
authored
Improve prompt error handling and relative path support (#153)
* Initial plan * Add path resolution and error handling for prompt file path parameters - Add resolvePromptFilePath() utility that resolves relative paths against workspace root and returns user-friendly warnings for invalid paths - Update all 9 prompt handlers with file path parameters to resolve paths and embed warnings in the response instead of throwing protocol errors - Add integration test in mcp-test-suite.js for explain_codeql_query with invalid path - Add file-based integration test fixtures in primitives/prompts/ - Add 10 unit tests for resolvePromptFilePath and handler path resolution Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> * Address code review: add advisory comment on existsSync, improve test assertions Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> * plan: add extension integration test for prompt error handling Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> * Add extension integration test for MCP prompt error handling with invalid paths Add mcp-prompt-e2e.integration.test.ts to the VS Code extension test suite that spawns the real MCP server inside the Extension Development Host and verifies that prompts return user-friendly warnings (not protocol errors) when given invalid file paths: - explain_codeql_query with nonexistent relative path returns "does not exist" - explain_codeql_query with valid absolute path returns no warning - document_codeql_query with nonexistent path returns "does not exist" - Server lists prompts including explain_codeql_query Register the new test in esbuild.config.js so it is bundled for CI. Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> * Initial plan * fix: return inline errors for invalid prompt inputs The MCP SDK validates prompt arguments before calling handlers. When validation fails (e.g. unsupported language like "rust"), the SDK throws McpError(-32602) with a raw JSON dump — poor UX for VS Code slash commands. Server-side changes (server/src/prompts/workflow-prompts.ts): - Add toPermissiveShape() to widen z.enum() to z.string() in registered schemas so the SDK never rejects user input at the protocol layer - Add createSafePromptHandler() to wrap all 11 prompt handlers with: (1) strict Zod validation that returns user-friendly inline errors (2) exception recovery that catches handler crashes gracefully - Add formatValidationError() to convert ZodError into actionable markdown with field names, received values, and valid options listed Integration tests (extensions/vscode/test/suite/mcp-prompt-e2e.integration.test.ts): - Expand from 4 to 15 e2e tests covering all prompts with file-path params - Add test for invalid language returning inline error (not protocol error) - Add test for path traversal detection - Add tests for all-optional prompts with empty args Unit tests (server/test/src/prompts/workflow-prompts.test.ts): - Add 17 new tests for toPermissiveShape, formatValidationError, createSafePromptHandler, and handler-level invalid argument handling - Update schema-to-registration consistency tests for permissive shapes * fix: promote key prompt parameters from optional to required Make databasePath, language, queryPath, and sarifPath required where the prompt depends on them. VS Code now correctly marks these fields as required in the slash-command input dialog. * Addres PR review feedback * Sync server/dist/codeql-development-mcp-server.js.map * [UPDATE PRIMITIVE] Fix path traversal check, add prompt fixture runner, fix comment mismatch (#155) * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Nathan Randall <70299490+data-douser@users.noreply.github.com> * Enforce tidy and rebuild server dist * address PR review comments for prompt path handling - Treat path traversal as hard failure: return blocked=true with empty resolvedPath instead of leaking the outside-root absolute path - Handle file:// URIs in resolvePromptFilePath via fileURLToPath so all callers (queryPath, workspaceUri, etc.) get consistent URI handling - Replace synchronous existsSync with async fs/promises.access to avoid blocking the event loop on the prompt hot path - Fix integration test success condition to fail when no tests executed - Make VS Code e2e invalid-language test deterministic (assert inline validation instead of accepting both throw and inline error) - Fix misleading test name "should return the original path" to match actual behavior (resolves relative to workspace root) * [WIP] Improve prompt error handling and relative path support (#157) * Initial plan * Fix prompt test runner: propagate failures and check sessions[].expectedContentPatterns Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/2840e119-cc2c-4f60-91ce-ba685ce25b5c --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> * More fixes for PR review feedback Improves end-to-end extension integration tests such that the 'CODEQL_MCP_WORKSPACE' env var is passed to StdioClientTransport, which ensures that the server's relative-path resolution uses a deterministic diretory instead of depending upon the Extension Host's working directory. * [UPDATE PRIMITIVE] Fix markdown injection and platform-dependent path tests in prompt handlers (#162) * Initial plan * Fix markdown injection and platform-dependent path tests in prompt handlers - Add sanitizeForInlineCode() helper to escape backticks and newlines in user-supplied values embedded in markdown code spans - Apply sanitizer to resolvePromptFilePath 'does not exist' warning (filePath and absolutePath) - Apply sanitizer to formatValidationError issue.received display - Fix POSIX path separator assumptions in tests: use basename only ('mydb', 'database') - Rename createSafePromptHandler tests to clarify they validate the handler wrapper, not MCP SDK validation Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/2660567b-5822-4505-91c2-37fe3ef00f4e * Replace sanitizeForInlineCode with CommonMark-compliant markdownInlineCode - markdownInlineCode() uses a fence length = maxRun+1 per CommonMark spec, preserving the original string (no information loss from backtick→apostrophe) - Normalises CR/CRLF to LF before wrapping (inline spans can't span lines) - Export markdownInlineCode for testability - Add 6 unit tests for markdownInlineCode (plain text, single/double backtick, CRLF normalisation, backtick-only values) - Add regression test for formatValidationError with backtick in received value - Add regression test for resolvePromptFilePath warning with backtick in path Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/ec7c534b-93ac-40e5-bcb6-023bc7496940 * Fix markdownInlineCode to replace newlines with spaces for single-line output Replace \r\n, \r, and \n with a space (not just normalize CRLF to LF) so the returned inline code span never contains a literal newline character. Update docstring and test to reflect space-replacement behavior. Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/87cfd54e-9d66-4871-a581-601aff3c6c8d --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com> * Add 'actions/cache@v5' for VS Code integration test download Adds a common cache for actions workflows that download the latest instance of VS Code for integration testing purposes. * Address latest feedback for PR #153 * Add retries to download-vscode.js script * fix: resilient VS Code download for integration tests - Add retry logic (3 attempts) with backoff to download-vscode.js - Handle unhandled promise rejections from @vscode/test-electron stream errors - Clean partial downloads between retries to avoid checksum mismatches - Default to 'stable' version to match .vscode-test.mjs configuration - Chain download:vscode before vscode-test in test:integration script - Cache VS Code downloads in build-and-test-extension.yml and copilot-setup-steps.yml - Increase download timeout from 15s (default) to 120s Fixes intermittent CI failures from ECONNRESET and empty-file checksum errors when downloading VS Code for Extension Host integration tests. * Address PR #153 review comments - resolvePromptFilePath: only enforce workspace-root boundary for relative path inputs; absolute paths pass through since users legitimately reference databases/queries outside the workspace - download-vscode.js: use fileURLToPath() instead of URL.pathname for cross-platform compatibility - integration-test-runner.js: track execution counts separately from pass/fail; runWorkflowIntegrationTests and runPromptIntegrationTests return { executed, passed } objects - Add test for absolute paths outside workspace being allowed * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Nathan Randall <70299490+data-douser@users.noreply.github.com> * fix: address PR #153 review comments for path resolution - Remove absolute path from 'does not exist' warning to avoid leaking local machine paths into prompt context sent to LLMs - Add blockedPathError() helper and check blocked flag at all 12 resolvePromptFilePath call sites so traversal attempts short-circuit with a clear inline error instead of silently proceeding - Add tests for both behaviors --------- Signed-off-by: Nathan Randall <70299490+data-douser@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 504152c commit 23ffb65

File tree

17 files changed

+2759
-810
lines changed

17 files changed

+2759
-810
lines changed

‎.github/workflows/build-and-test-extension.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,16 @@ jobs:
6363
add-to-path: 'true'
6464
install-language-runtimes: 'false'
6565

66+
- name: Cache VS Code for integration tests
67+
uses: actions/cache@v5
68+
with:
69+
key: vscode-test-${{ runner.os }}-stable
70+
path: extensions/vscode/.vscode-test
71+
72+
- name: Download VS Code for integration tests
73+
working-directory: extensions/vscode
74+
run: npm run download:vscode
75+
6676
- name: Run Extension Host integration tests
6777
working-directory: extensions/vscode
6878
run: xvfb-run -a npm run test:integration

‎.github/workflows/copilot-setup-steps.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,12 @@ jobs:
7777
npm run bundle
7878
npm run bundle:server
7979
80+
- name: Copilot Setup - Cache VS Code for integration tests
81+
uses: actions/cache@v5
82+
with:
83+
key: vscode-test-${{ runner.os }}-stable
84+
path: extensions/vscode/.vscode-test
85+
8086
- name: Copilot Setup - Download VS Code for integration tests
8187
working-directory: extensions/vscode
8288
run: npm run download:vscode

‎client/integration-tests/primitives/prompts/explain_codeql_query/invalid_query_path/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Integration Test: explain_codeql_query / invalid_query_path
2+
3+
## Purpose
4+
5+
Verify that the `explain_codeql_query` prompt returns a helpful error message
6+
when the user provides a `queryPath` that does not exist on disk.
7+
8+
## Expected Behavior
9+
10+
- The prompt handler resolves the relative path against the workspace root.
11+
- When the resolved path does not exist, the prompt returns a warning in the
12+
response messages rather than throwing a raw MCP protocol error.
13+
- The warning message should mention the file was not found so the user can
14+
correct the path.
15+
16+
## Parameters
17+
18+
| Parameter | Value | Notes |
19+
| -------------- | ------------------------------ | ------------------ |
20+
| `databasePath` | `nonexistent/path/to/database` | Does **not** exist |
21+
| `queryPath` | `nonexistent/path/to/query.ql` | Does **not** exist |
22+
| `language` | `javascript` | Valid language |

‎client/integration-tests/primitives/prompts/explain_codeql_query/invalid_query_path/after/monitoring-state.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"sessions": [
3+
{
4+
"expectedContentPatterns": [
5+
"does not exist"
6+
]
7+
}
8+
]
9+
}

‎client/integration-tests/primitives/prompts/explain_codeql_query/invalid_query_path/before/monitoring-state.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"sessions": [],
3+
"parameters": {
4+
"databasePath": "nonexistent/path/to/database",
5+
"queryPath": "nonexistent/path/to/query.ql",
6+
"language": "javascript"
7+
}
8+
}

‎client/integration-tests/primitives/tools/codeql_bqrs_interpret/sarif_format/after/results.sarif‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{"$schema":"https://json.schemastore.org/sarif-2.1.0.json","version":"2.1.0","runs":[{"tool":{"driver":{"name":"CodeQL","organization":"GitHub","semanticVersion":"2.24.3","rules":[{"id":"test/query","name":"test/query","shortDescription":{"text":"ExampleQuery1"},"fullDescription":{"text":"Example query for integration testing of the codeql_test_extract MCP server tool."},"defaultConfiguration":{"enabled":true,"level":"warning"},"help":{"text":"# Query Help for JavaScript ExampleQuery1\n\nTODO\n","markdown":"# Query Help for JavaScript ExampleQuery1\n\nTODO\n"},"properties":{"tags":["mcp-integration-tests"],"description":"Example query for integration testing of the codeql_test_extract MCP server tool.","id":"test/query","kind":"problem","name":"ExampleQuery1","precision":"medium","problem.severity":"warning"}}]},"extensions":[{"name":"mcp-client-integration-tests-static-javascript-src","semanticVersion":"0.0.1+fe0e7d2a7059ebb6c6075ff8eaea04f382747656","locations":[{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/src/","description":{"text":"The QL pack root directory."},"properties":{"tags":["CodeQL/LocalPackRoot"]}},{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/src/codeql-pack.yml","description":{"text":"The QL pack definition file."},"properties":{"tags":["CodeQL/LocalPackDefinitionFile"]}}]},{"name":"codeql/javascript-all","semanticVersion":"2.6.11+ce9c8e6e9fd41ef0967b13849bb6ae2183caf9ad","locations":[{"uri":"file:///home/runner/.codeql/packages/codeql/javascript-all/2.6.11/","description":{"text":"The QL pack root directory."},"properties":{"tags":["CodeQL/LocalPackRoot"]}},{"uri":"file:///home/runner/.codeql/packages/codeql/javascript-all/2.6.11/qlpack.yml","description":{"text":"The QL pack definition file."},"properties":{"tags":["CodeQL/LocalPackDefinitionFile"]}}]},{"name":"codeql/threat-models","semanticVersion":"1.0.31+ce9c8e6e9fd41ef0967b13849bb6ae2183caf9ad","locations":[{"uri":"file:///home/runner/.codeql/packages/codeql/threat-models/1.0.31/","description":{"text":"The QL pack root directory."},"properties":{"tags":["CodeQL/LocalPackRoot"]}},{"uri":"file:///home/runner/.codeql/packages/codeql/threat-models/1.0.31/qlpack.yml","description":{"text":"The QL pack definition file."},"properties":{"tags":["CodeQL/LocalPackDefinitionFile"]}}]}]},"artifacts":[{"location":{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/test/ExampleQuery1/ExampleQuery1.js","index":0}}],"results":[{"ruleId":"test/query","ruleIndex":0,"rule":{"id":"test/query","index":0},"message":{"text":"Example test code file found for codeql_test_extract example query."},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/test/ExampleQuery1/ExampleQuery1.js","index":0}}}]}],"columnKind":"utf16CodeUnits","properties":{"semmle.formatSpecifier":"sarif-latest"}}]}
1+
{"$schema":"https://json.schemastore.org/sarif-2.1.0.json","version":"2.1.0","runs":[{"tool":{"driver":{"name":"CodeQL","organization":"GitHub","semanticVersion":"2.25.0","rules":[{"id":"test/query","name":"test/query","shortDescription":{"text":"ExampleQuery1"},"fullDescription":{"text":"Example query for integration testing of the codeql_test_extract MCP server tool."},"defaultConfiguration":{"enabled":true,"level":"warning"},"help":{"text":"# Query Help for JavaScript ExampleQuery1\n\nTODO\n","markdown":"# Query Help for JavaScript ExampleQuery1\n\nTODO\n"},"properties":{"tags":["mcp-integration-tests"],"description":"Example query for integration testing of the codeql_test_extract MCP server tool.","id":"test/query","kind":"problem","name":"ExampleQuery1","precision":"medium","problem.severity":"warning"}}]},"extensions":[{"name":"mcp-client-integration-tests-static-javascript-src","semanticVersion":"0.0.1+fe0e7d2a7059ebb6c6075ff8eaea04f382747656","locations":[{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/src/","description":{"text":"The QL pack root directory."},"properties":{"tags":["CodeQL/LocalPackRoot"]}},{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/src/codeql-pack.yml","description":{"text":"The QL pack definition file."},"properties":{"tags":["CodeQL/LocalPackDefinitionFile"]}}]},{"name":"codeql/javascript-all","semanticVersion":"2.6.11+ce9c8e6e9fd41ef0967b13849bb6ae2183caf9ad","locations":[{"uri":"file:///home/runner/.codeql/packages/codeql/javascript-all/2.6.11/","description":{"text":"The QL pack root directory."},"properties":{"tags":["CodeQL/LocalPackRoot"]}},{"uri":"file:///home/runner/.codeql/packages/codeql/javascript-all/2.6.11/qlpack.yml","description":{"text":"The QL pack definition file."},"properties":{"tags":["CodeQL/LocalPackDefinitionFile"]}}]},{"name":"codeql/threat-models","semanticVersion":"1.0.31+ce9c8e6e9fd41ef0967b13849bb6ae2183caf9ad","locations":[{"uri":"file:///home/runner/.codeql/packages/codeql/threat-models/1.0.31/","description":{"text":"The QL pack root directory."},"properties":{"tags":["CodeQL/LocalPackRoot"]}},{"uri":"file:///home/runner/.codeql/packages/codeql/threat-models/1.0.31/qlpack.yml","description":{"text":"The QL pack definition file."},"properties":{"tags":["CodeQL/LocalPackDefinitionFile"]}}]}]},"artifacts":[{"location":{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/test/ExampleQuery1/ExampleQuery1.js","index":0}}],"results":[{"ruleId":"test/query","ruleIndex":0,"rule":{"id":"test/query","index":0},"message":{"text":"Example test code file found for codeql_test_extract example query."},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"file:///home/runner/work/codeql-development-mcp-server/codeql-development-mcp-server/client/integration-tests/static/javascript/test/ExampleQuery1/ExampleQuery1.js","index":0}}}]}],"columnKind":"utf16CodeUnits","properties":{"semmle.formatSpecifier":"sarif-latest"}}]}

‎client/src/lib/integration-test-runner.js‎

Lines changed: 196 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -226,10 +226,45 @@ export class IntegrationTestRunner {
226226

227227
this.logger.log(`Completed ${totalIntegrationTests} tool-specific integration tests`);
228228

229+
// Track execution counts and pass/fail separately.
230+
// *Count* tracks whether the suite discovered & ran tests.
231+
// *Succeeded* tracks whether those tests passed.
232+
const toolTestsExecuted = totalIntegrationTests;
233+
const toolTestsPassed = totalIntegrationTests > 0;
234+
229235
// Also run workflow integration tests
230-
await this.runWorkflowIntegrationTests(baseDir);
236+
const { executed: workflowTestsExecuted, passed: workflowTestsPassed } =
237+
await this.runWorkflowIntegrationTests(baseDir);
238+
if (!workflowTestsPassed) {
239+
this.logger.logTest(
240+
"Workflow integration tests",
241+
false,
242+
new Error("Workflow integration tests did not complete successfully")
243+
);
244+
}
245+
246+
// Also run prompt integration tests
247+
const { executed: promptTestsExecuted, passed: promptTestsPassed } =
248+
await this.runPromptIntegrationTests(baseDir);
249+
if (!promptTestsPassed) {
250+
this.logger.logTest(
251+
"Prompt integration tests",
252+
false,
253+
new Error("Prompt integration tests did not complete successfully")
254+
);
255+
}
256+
257+
const totalTestsExecuted = toolTestsExecuted + workflowTestsExecuted + promptTestsExecuted;
258+
259+
if (totalTestsExecuted === 0) {
260+
this.logger.log(
261+
"No integration tests were executed across tool, workflow, or prompt suites.",
262+
"ERROR"
263+
);
264+
return false;
265+
}
231266

232-
return totalIntegrationTests > 0;
267+
return toolTestsPassed && workflowTestsPassed && promptTestsPassed;
233268
} catch (error) {
234269
this.logger.log(`Error running integration tests: ${error.message}`, "ERROR");
235270
return false;
@@ -1095,6 +1130,161 @@ export class IntegrationTestRunner {
10951130
return params;
10961131
}
10971132

1133+
/**
1134+
* Run prompt-level integration tests.
1135+
* Discovers test fixtures under `integration-tests/primitives/prompts/`
1136+
* and calls `client.getPrompt()` for each, validating the response.
1137+
*/
1138+
async runPromptIntegrationTests(baseDir) {
1139+
try {
1140+
this.logger.log("Discovering and running prompt integration tests...");
1141+
1142+
const promptTestsDir = path.join(baseDir, "..", "integration-tests", "primitives", "prompts");
1143+
1144+
if (!fs.existsSync(promptTestsDir)) {
1145+
this.logger.log("No prompt integration tests directory found", "INFO");
1146+
return {
1147+
executed: 0,
1148+
passed: true
1149+
};
1150+
}
1151+
1152+
// Get list of available prompts from the server
1153+
const response = await this.client.listPrompts();
1154+
const prompts = response.prompts || [];
1155+
const promptNames = prompts.map((p) => p.name);
1156+
1157+
this.logger.log(`Found ${promptNames.length} prompts available on server`);
1158+
1159+
// Discover prompt test directories (each subdirectory = one prompt name)
1160+
const promptDirs = fs
1161+
.readdirSync(promptTestsDir, { withFileTypes: true })
1162+
.filter((dirent) => dirent.isDirectory())
1163+
.map((dirent) => dirent.name);
1164+
1165+
this.logger.log(
1166+
`Found ${promptDirs.length} prompt test directories: ${promptDirs.join(", ")}`
1167+
);
1168+
1169+
let totalPromptTests = 0;
1170+
let allPromptTestsPassed = true;
1171+
for (const promptName of promptDirs) {
1172+
if (!promptNames.includes(promptName)) {
1173+
this.logger.log(`Skipping ${promptName} - prompt not found on server`, "WARN");
1174+
continue;
1175+
}
1176+
1177+
const promptDir = path.join(promptTestsDir, promptName);
1178+
const testCases = fs
1179+
.readdirSync(promptDir, { withFileTypes: true })
1180+
.filter((dirent) => dirent.isDirectory())
1181+
.map((dirent) => dirent.name);
1182+
1183+
this.logger.log(`Running ${testCases.length} test case(s) for prompt ${promptName}`);
1184+
1185+
for (const testCase of testCases) {
1186+
const passed = await this.runSinglePromptIntegrationTest(promptName, testCase, promptDir);
1187+
totalPromptTests++;
1188+
if (!passed) {
1189+
allPromptTestsPassed = false;
1190+
}
1191+
}
1192+
}
1193+
1194+
this.logger.log(`Total prompt integration tests executed: ${totalPromptTests}`);
1195+
return {
1196+
executed: totalPromptTests,
1197+
passed: totalPromptTests > 0 ? allPromptTestsPassed : true
1198+
};
1199+
} catch (error) {
1200+
this.logger.log(`Error running prompt integration tests: ${error.message}`, "ERROR");
1201+
return { executed: 0, passed: false };
1202+
}
1203+
}
1204+
1205+
/**
1206+
* Run a single prompt integration test.
1207+
*
1208+
* Reads parameters from `before/monitoring-state.json`, calls `getPrompt()`,
1209+
* and validates that the response contains messages (no protocol-level error).
1210+
*/
1211+
async runSinglePromptIntegrationTest(promptName, testCase, promptDir) {
1212+
const testName = `prompt:${promptName}/${testCase}`;
1213+
this.logger.log(`\nRunning prompt integration test: ${testName}`);
1214+
1215+
try {
1216+
const testCaseDir = path.join(promptDir, testCase);
1217+
const beforeDir = path.join(testCaseDir, "before");
1218+
const afterDir = path.join(testCaseDir, "after");
1219+
1220+
// Validate test structure
1221+
if (!fs.existsSync(beforeDir)) {
1222+
this.logger.logTest(testName, false, "Missing before directory");
1223+
return false;
1224+
}
1225+
1226+
if (!fs.existsSync(afterDir)) {
1227+
this.logger.logTest(testName, false, "Missing after directory");
1228+
return false;
1229+
}
1230+
1231+
// Load parameters from before/monitoring-state.json
1232+
const monitoringStatePath = path.join(beforeDir, "monitoring-state.json");
1233+
if (!fs.existsSync(monitoringStatePath)) {
1234+
this.logger.logTest(testName, false, "Missing before/monitoring-state.json");
1235+
return false;
1236+
}
1237+
1238+
const monitoringState = JSON.parse(fs.readFileSync(monitoringStatePath, "utf8"));
1239+
const params = monitoringState.parameters || {};
1240+
resolvePathPlaceholders(params, this.logger);
1241+
1242+
// Call the prompt
1243+
this.logger.log(`Calling prompt ${promptName} with params: ${JSON.stringify(params)}`);
1244+
1245+
const result = await this.client.getPrompt({
1246+
name: promptName,
1247+
arguments: params
1248+
});
1249+
1250+
// Validate that the response contains messages (no raw protocol error)
1251+
const hasMessages = result.messages && result.messages.length > 0;
1252+
if (!hasMessages) {
1253+
this.logger.logTest(testName, false, "Expected messages in prompt response");
1254+
return false;
1255+
}
1256+
1257+
// If the after/monitoring-state.json has expected content checks, validate
1258+
const afterMonitoringPath = path.join(afterDir, "monitoring-state.json");
1259+
if (fs.existsSync(afterMonitoringPath)) {
1260+
const afterState = JSON.parse(fs.readFileSync(afterMonitoringPath, "utf8"));
1261+
1262+
// Support both top-level expectedContentPatterns and sessions[].expectedContentPatterns
1263+
const sessions = afterState.sessions || [];
1264+
const topLevelPatterns = afterState.expectedContentPatterns || [];
1265+
const sessionPatterns =
1266+
sessions.length > 0 ? sessions[0].expectedContentPatterns || [] : [];
1267+
const expectedPatterns = topLevelPatterns.length > 0 ? topLevelPatterns : sessionPatterns;
1268+
1269+
if (expectedPatterns.length > 0) {
1270+
const text = result.messages[0]?.content?.text || "";
1271+
for (const pattern of expectedPatterns) {
1272+
if (!text.includes(pattern)) {
1273+
this.logger.logTest(testName, false, `Expected response to contain "${pattern}"`);
1274+
return false;
1275+
}
1276+
}
1277+
}
1278+
}
1279+
1280+
this.logger.logTest(testName, true, `Prompt returned ${result.messages.length} message(s)`);
1281+
return true;
1282+
} catch (error) {
1283+
this.logger.logTest(testName, false, `Error: ${error.message}`);
1284+
return false;
1285+
}
1286+
}
1287+
10981288
/**
10991289
* Run workflow-level integration tests
11001290
* These tests validate complete workflows rather than individual tools
@@ -1107,7 +1297,7 @@ export class IntegrationTestRunner {
11071297

11081298
if (!fs.existsSync(workflowTestsDir)) {
11091299
this.logger.log("No workflow integration tests directory found", "INFO");
1110-
return true;
1300+
return { executed: 0, passed: true };
11111301
}
11121302

11131303
// Discover workflow test directories
@@ -1118,7 +1308,7 @@ export class IntegrationTestRunner {
11181308

11191309
if (workflowDirs.length === 0) {
11201310
this.logger.log("No workflow test directories found", "INFO");
1121-
return true;
1311+
return { executed: 0, passed: true };
11221312
}
11231313

11241314
this.logger.log(`Found ${workflowDirs.length} workflow test(s): ${workflowDirs.join(", ")}`);
@@ -1131,10 +1321,10 @@ export class IntegrationTestRunner {
11311321
}
11321322

11331323
this.logger.log(`Total workflow integration tests executed: ${totalWorkflowTests}`);
1134-
return true;
1324+
return { executed: totalWorkflowTests, passed: true };
11351325
} catch (error) {
11361326
this.logger.log(`Error running workflow integration tests: ${error.message}`, "ERROR");
1137-
return false;
1327+
return { executed: 0, passed: false };
11381328
}
11391329
}
11401330

‎client/src/lib/mcp-test-suite.js‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ export class MCPTestSuite {
2222
await this.testReadResource();
2323
await this.testListPrompts();
2424
await this.testGetPrompt();
25+
await this.testGetPromptWithInvalidPath();
2526
}
2627

2728
/**
@@ -132,6 +133,59 @@ export class MCPTestSuite {
132133
}
133134
}
134135

136+
/**
137+
* Test that prompts handle invalid file paths gracefully.
138+
*
139+
* The explain_codeql_query prompt requires a `queryPath` parameter.
140+
* When given a nonexistent path it should return a user-friendly error
141+
* message inside the prompt response rather than throwing a raw MCP
142+
* protocol error.
143+
*/
144+
async testGetPromptWithInvalidPath() {
145+
try {
146+
this.logger.log("Testing prompt error handling with invalid file path...");
147+
148+
const result = await this.client.getPrompt({
149+
name: "explain_codeql_query",
150+
arguments: {
151+
databasePath: "nonexistent/path/to/database",
152+
queryPath: "nonexistent/path/to/query.ql",
153+
language: "javascript"
154+
}
155+
});
156+
157+
// The prompt should return messages (not throw) even for an invalid path
158+
const hasMessages = result.messages && result.messages.length > 0;
159+
if (!hasMessages) {
160+
this.logger.logTest(
161+
"Get Prompt with Invalid Path (explain_codeql_query)",
162+
false,
163+
"Expected messages in response"
164+
);
165+
return false;
166+
}
167+
168+
// The response should contain a human-readable error about the invalid path
169+
const text = result.messages[0]?.content?.text || "";
170+
const mentionsPathError =
171+
text.includes("does not exist") ||
172+
text.includes("not found") ||
173+
text.includes("could not be found") ||
174+
text.includes("File not found") ||
175+
text.includes("⚠");
176+
177+
this.logger.log(`Prompt response mentions path error: ${mentionsPathError}`);
178+
this.logger.logTest("Get Prompt with Invalid Path (explain_codeql_query)", mentionsPathError);
179+
180+
return mentionsPathError;
181+
} catch (error) {
182+
// If the server throws instead of returning a message this test fails,
183+
// which is the exact behaviour we want to fix.
184+
this.logger.logTest("Get Prompt with Invalid Path (explain_codeql_query)", false, error);
185+
return false;
186+
}
187+
}
188+
135189
/**
136190
* Test listing prompts
137191
*/

0 commit comments

Comments
 (0)