Use a POST-only transport for external MCP connections (fix list_tools hang)

[goelise]

Draft. Opening this to discuss the approach. The core fix and unit tests are here; it has not yet been validated against a live external connection (needs an OBO user token), and the agent-facing piece (databricks_openai McpServer) is intentionally not included yet — see "Scope" below.

Problem

DatabricksMCPClient.list_tools() / call_tool() hang (~20s, then fail) when the server URL is an external/managed connection at /api/2.0/mcp/external/<connection> (e.g. the managed Atlassian connection). The same connection works from the AI Playground and from a hand-rolled POST-only client, which makes this confusing.

Root cause — transport mismatch, not a broken proxy

External connections are fronted by a POST-only JSON-REST bridge: it rejects any non-POST method with 405 Method Not Allowed and replies with application/json, never text/event-stream. There is no GET endpoint, no SSE stream, and no server-initiated messages. (This is spec-legal — the GET SSE stream in Streamable HTTP is optional.)

DatabricksMCPClient (and databricks_openai.agents.McpServer) go through the mcp SDK's streamablehttp_client, a full Streamable HTTP client that, on connect, opens a background GET SSE stream for server→client messages. Against the bridge that GET gets 405 and the stream task dies; a later request whose response handling expects coordination with that (now-dead) stream then blocks until timeout. This is the documented upstream issue modelcontextprotocol/python-sdk#1941 ("hangs indefinitely when connecting to POST-only MCP servers", reported against GitHub's POST-only MCP server — same class of server as the bridge).

Two adjacent factors compound it: the initialized-notification POST can be treated as fatal by default (openai-agents made tolerance opt-in in #2765), and the SDK advertises a very new LATEST_PROTOCOL_VERSION.

Fix in this PR

Route EXTERNAL_MCP URLs (already classified in MCP_URL_PATTERNS) through a new PostOnlyMCPClient that speaks plain request/response JSON-RPC over POST: initialize → notifications/initialized → tools/list / tools/call, parsing the JSON-RPC frame as JSON or, defensively, a single SSE data: frame. It returns the same mcp.types.Tool / CallToolResult objects, so it's a drop-in for the external branch. Managed servers (Genie, UC functions, vector search) are unchanged — they keep using streamablehttp_client.

• databricks_mcp/external_client.py — new PostOnlyMCPClient.
• databricks_mcp/mcp.py — _get_tools_async / _call_tools_async branch on EXTERNAL_MCP.
• Unit tests asserting external → PostOnlyMCPClient and managed → streamablehttp_client (existing tests unchanged).

Scope / open questions for reviewers

1. databricks_openai.agents.McpServer is not addressed here. It subclasses MCPServerStreamableHttp, whose whole design is the streamable transport, so a POST-only server doesn't fit create_streams. Fixing the agent path needs a separate MCPServer subclass over POST-only HTTP. Want guidance on whether to add that here or in a follow-up, and where it should live.
2. vs. upstream fix. The hang is ultimately mcp SDK #1941 (open PR #1943). Is a Databricks-side POST-only transport the preferred path, or should we wait on/track upstream?
3. Protocol version is pinned to 2025-03-26 (what the bridge negotiates). Reviewers may prefer to negotiate from the initialize response instead.

Validation status

• Unit tests added; branching logic verified locally (external → POST-only, managed → streamable).
• Not yet run against a live external connection over OBO — will validate before marking ready.

Reference POST-only client originates from sunish.sheth's dais-debug app.