Skip to content

Evaluate migrating tests/support/mcp.py to the official mcp PyPI SDK #74

Description

@WilliamBergamin

Background

The eval suite talks to the Slack MCP server (https://mcp.slack.com/mcp) through a hand-rolled, stdlib-only Streamable HTTP JSON-RPC client at tests/support/mcp.py (MCPClient). It's ~130 lines, synchronous, and dependency-free, which suits the sync pytest suite well.

This issue tracks evaluating whether to replace it with the official mcp PyPI SDK (streamablehttp_client + ClientSession).

Spun out of PR #70 review: the question came up while investigating a 401 Unauthorized failure. Worth noting up front — the 401 was an expired/invalid token, not a client bug, so the SDK migration would not have prevented it. That failure is being addressed separately (clearer 401/403 error message + token docs).

Trade-offs

For migrating:

  • Spec-compliant transport, maintained upstream (handles protocol/version drift, SSE edge cases, session lifecycle for us).
  • Less bespoke code to maintain.

Against / cost:

  • The SDK is async (asyncio); our pytest eval suite is synchronous, so we'd need an asyncio.run(...) wrapper around tool-listing.
  • Adds a runtime dependency (mcp, which pulls in httpx/anyio/pydantic) to a suite that is currently stdlib-only for this path.
  • Does not address token expiry — the root cause of the reported 401.

Suggested outcome

Prototype the tool-listing call (list_tools) against mcp's streamablehttp_client(url, headers={"Authorization": f"Bearer {token}"}) + ClientSession, wrapped for sync use, and compare against the current MCPClient on reliability and complexity. Decide keep-vs-migrate from there.

Co-Authored-By: Claude svc-devxp-claude@slack-corp.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions