test(scorecard): add MCP integration coverage for scorecard tools by HusneShabbir · Pull Request #3664 · redhat-developer/rhdh-plugins

HusneShabbir · 2026-07-02T08:50:14Z

Summary

Adds MCP integration coverage for Scorecard backend tools with a reusable test helper.

Resolves:
https://redhat.atlassian.net/browse/RHIDP-14592

Test Coverage

`scorecard.list-metrics`

Tool is discoverable via tools/list
Tool metadata is validated (readOnlyHint, destructiveHint shape)
Returns configured metrics on success
Applies conditional permission filtering (returns only allowed metric IDs)
Not exposed when scorecard is excluded from backend.actions.pluginSources

`scorecard.get-entity-metrics`

Returns metrics for a valid existing entity
Returns error when entity is not found
Returns error when entity access is denied
Returns validation error when entityRef is missing
Returns validation error when entityRef type is invalid

Test Infrastructure

Introduced reusable MCP integration helpers in packages/backend/src/__fixtures__/mcpTestUtils.ts
Added backend script: test:integration

Verification

yarn --cwd workspaces/scorecard/packages/backend run test src/mcp-tools.integration.test.ts --watch=false
Result: 10/10 tests passing

Add backend-level MCP integration tests for list-metrics and get-entity-metrics, including visibility, permission, validation, and error-path checks. Co-authored-by: Cursor <cursoragent@cursor.com>

rhdh-gh-app · 2026-07-02T08:50:40Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
backend	workspaces/scorecard/packages/backend	none	`v0.0.0`

fullsend-ai-review · 2026-07-02T08:53:14Z

🤖 Finished Review · ✅ Success · Started 8:53 AM UTC · Completed 9:06 AM UTC
Commit: c7f0beb · View workflow run →

codecov · 2026-07-02T08:55:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.09%. Comparing base (904c783) to head (3a2e1ee).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3664      +/-   ##
==========================================
- Coverage   54.31%   54.09%   -0.23%     
==========================================
  Files        2349     2340       -9     
  Lines       89701    89309     -392     
  Branches    25127    24958     -169     
==========================================
- Hits        48722    48312     -410     
- Misses      40685    40709      +24     
+ Partials      294      288       -6

Flag	Coverage Δ		*Carryforward flag
adoption-insights	`83.70% <ø> (ø)`		Carriedforward from 4a5d9ed
ai-integrations	`68.16% <ø> (-0.19%)`	⬇️	Carriedforward from 4a5d9ed
app-defaults	`69.79% <ø> (ø)`		Carriedforward from 4a5d9ed
augment	`46.39% <ø> (ø)`		Carriedforward from 4a5d9ed
boost	`74.68% <ø> (ø)`		Carriedforward from 4a5d9ed
bulk-import	`72.46% <ø> (ø)`		Carriedforward from 4a5d9ed
cost-management	`14.10% <ø> (ø)`		Carriedforward from 4a5d9ed
dcm	`61.81% <ø> (ø)`		Carriedforward from 4a5d9ed
extensions	`61.53% <ø> (ø)`		Carriedforward from 4a5d9ed
global-floating-action-button	`71.18% <ø> (ø)`		Carriedforward from 4a5d9ed
global-header	`59.71% <ø> (ø)`		Carriedforward from 4a5d9ed
homepage	`48.18% <ø> (-2.05%)`	⬇️	Carriedforward from 4a5d9ed
install-dynamic-plugins	`56.77% <ø> (ø)`		Carriedforward from 4a5d9ed
konflux	`91.49% <ø> (ø)`		Carriedforward from 4a5d9ed
lightspeed	`68.69% <ø> (-0.13%)`	⬇️	Carriedforward from 4a5d9ed
mcp-integrations	`85.46% <ø> (ø)`		Carriedforward from 4a5d9ed
orchestrator	`37.25% <ø> (-2.64%)`	⬇️	Carriedforward from 4a5d9ed
quickstart	`65.63% <ø> (ø)`		Carriedforward from 4a5d9ed
sandbox	`79.56% <ø> (ø)`		Carriedforward from 4a5d9ed
scorecard	`82.67% <ø> (ø)`
theme	`61.26% <ø> (ø)`		Carriedforward from 4a5d9ed
translations	`7.25% <ø> (ø)`		Carriedforward from 4a5d9ed
x2a	`78.68% <ø> (ø)`		Carriedforward from 4a5d9ed

*This pull request uses carry forward flags. Click here to find out more.

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 904c783...3a2e1ee. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

fullsend-ai-review · 2026-07-02T09:05:53Z

Review

Findings

Medium

[dependency-placement] workspaces/scorecard/packages/backend/package.json — Test-only dependencies (@backstage/backend-plugin-api, @backstage/backend-test-utils, @backstage/plugin-catalog-node, @modelcontextprotocol/sdk) are added to dependencies instead of devDependencies. The mcp-integrations workspace's backend package.json places the exact same packages in devDependencies. @backstage/backend-test-utils in particular is explicitly a test utility package that should never be a production dependency.
Remediation: Move these four packages from dependencies to devDependencies.
[type-definition-pattern] workspaces/scorecard/packages/backend/src/__fixtures__/mcpTestUtils.ts:36 — The CallToolResult type is defined manually as a hand-written interface. The reference implementation in mcp-integrations derives it from the Zod schema (type CallToolResult = z.infer<typeof CallToolResultSchema>), which ties the type to the SDK's source of truth and avoids drift if the schema changes.
Remediation: Use z.infer<typeof CallToolResultSchema> instead of a hand-written type.

Low

[error-handling] workspaces/scorecard/packages/backend/src/__fixtures__/mcpTestUtils.ts:133 — parseCallToolOutput does not check the isError flag before attempting to parse content. If a tool call fails but returns text that happens to be valid JSON, it could be silently treated as a success. The test file mitigates this by checking result.isError before calling the appropriate parser, but a defensive check in the utility itself would be more robust.
[code-duplication] workspaces/scorecard/packages/backend/src/__fixtures__/mcpTestUtils.ts — The functions stripMarkdownJsonFence, parseCallToolOutput, closeMcpConnection, createMcpTransport, withMcpClient, getServerPort, and MCP_TRANSPORT_RECONNECTION_OPTIONS are nearly identical copies of code in workspaces/mcp-integrations/packages/backend/src/mcp-tools.integration.test.ts. Consider extracting a shared test utility package if more workspaces adopt MCP integration tests.

fullsend-ai-review

See the review comment for full details.

Move MCP test-only packages back to devDependencies and switch CallToolResult to z.infer from the SDK schema to match the reference implementation. Co-authored-by: Cursor <cursoragent@cursor.com>

sonarqubecloud · 2026-07-06T17:33:32Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

yangcao77

generally looks good.

The only thing is the zod/v4 import. it should be zod to match the workspace and the reference test. Everything else is good to go.

yangcao77 · 2026-07-07T20:25:20Z

+import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';
+import { CallToolResultSchema } from '@modelcontextprotocol/sdk/types.js';
+import type { Server } from 'node:http';
+import type { z } from 'zod/v4';


The scorecard codebase uses from zod, and the mcp-integrations reference test also uses from zod. The zod/v4 subpath is a zod 4.x feature; this workspace is on zod 3.x.

test(scorecard): add MCP integration coverage for scorecard tools

69e4685

Add backend-level MCP integration tests for list-metrics and get-entity-metrics, including visibility, permission, validation, and error-path checks. Co-authored-by: Cursor <cursoragent@cursor.com>

HusneShabbir requested review from a team, Eswaraiahsapram, christoph-jerolimov and its-mitesh-kumar as code owners July 2, 2026 08:50

github-actions Bot added the workspace/scorecard label Jul 2, 2026

HusneShabbir requested review from a team and yangcao77 July 2, 2026 08:50

HusneShabbir requested a review from jrichter1 July 2, 2026 08:50

fullsend-ai-review Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread workspaces/scorecard/packages/backend/src/__fixtures__/mcpTestUtils.ts Outdated

Comment thread workspaces/scorecard/packages/backend/src/__fixtures__/mcpTestUtils.ts

fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Jul 2, 2026

HusneShabbir and others added 2 commits July 6, 2026 22:56

fix(scorecard): align MCP test helper typing and deps

4a5d9ed

Move MCP test-only packages back to devDependencies and switch CallToolResult to z.infer from the SDK schema to match the reference implementation. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'main' into qe/scorecard-mcp-integration-tests

3a2e1ee

yangcao77 reviewed Jul 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(scorecard): add MCP integration coverage for scorecard tools#3664

test(scorecard): add MCP integration coverage for scorecard tools#3664
HusneShabbir wants to merge 3 commits into
redhat-developer:mainfrom
HusneShabbir:qe/scorecard-mcp-integration-tests

HusneShabbir commented Jul 2, 2026 •

edited

Loading

Uh oh!

rhdh-gh-app Bot commented Jul 2, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026

Uh oh!

fullsend-ai-review Bot left a comment

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 6, 2026

Uh oh!

yangcao77 left a comment

Uh oh!

yangcao77 Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

HusneShabbir commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Coverage

scorecard.list-metrics

scorecard.get-entity-metrics

Test Infrastructure

Verification

Uh oh!

rhdh-gh-app Bot commented Jul 2, 2026

Changed Packages

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026

Review

Findings

Medium

Low

Uh oh!

fullsend-ai-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 6, 2026

Quality Gate passed

Uh oh!

yangcao77 left a comment

Choose a reason for hiding this comment

Uh oh!

yangcao77 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HusneShabbir commented Jul 2, 2026 •

edited

Loading

`scorecard.list-metrics`

`scorecard.get-entity-metrics`

fullsend-ai-review Bot commented Jul 2, 2026 •

edited

Loading

codecov Bot commented Jul 2, 2026 •

edited

Loading