ROB-4013 MCP auth health check: auto-detect + enable GitHub context (supersedes HolmesGPT#2117) (HolmesGPT#2118)

alonelish · claude · RoiGlinik · web-flow · commit 5b2872c131ca · 2026-06-03T14:27:31.000+03:00
> Split: the `/api/chat` raw-bytes-body fix that previously lived here has moved to HolmesGPT#2120. This PR is now MCP-only. Supersedes HolmesGPT#2117 (originally by @aantn); its commits are included. ## Summary Makes MCP toolset authentication validation work end-to-end. **From HolmesGPT#2117 (@aantn) — `health_check_tool` foundation** - Adds an opt-in `health_check_tool` config field to MCP toolsets. MCP servers (e.g. GitHub) return their full tool list even with an invalid credential, so listing tools never proves auth. When set, the named read-only tool is invoked during prerequisite evaluation; a failure (e.g. 401) marks the toolset disabled instead of "connected". Wires `get_me` / `get_current_user` into the helm chart for the built-in GitHub / GitLab addons. **Follow-up (this PR)** 1. **Auto-detect the health check tool.** Toolset configs generated outside the helm template (e.g. Robusta's `custom_toolset.yaml`) don't set `health_check_tool`, so the check was skipped. When unset, Holmes now falls back to an allowlist of well-known read-only identity tools (`get_me`, `get_current_user`, `get_authenticated_user`, `whoami`) and invokes the first one the server exposes. Explicit config still takes precedence. 2. **Enable the GitHub `context` toolset by default.** `get_me` only exists when the GitHub MCP server's `context` toolset is enabled, but the chart default (`repos,issues,pull_requests,actions`) excluded it — so auto-detection found nothing. Default is now `repos,issues,pull_requests,actions,context` (read-only/lightweight). Adds a debug log when no identity tool is exposed, so a skipped check is diagnosable. 3. **Detailed failure messages.** Health-check failures now include the tool name and params (`health check tool '<name>' with params {} failed - ...`), per the toolset error-detail contract (CodeRabbit). ## Testing `tests/test_mcp_toolset.py`: health-check, auto-detect, skipped-check-logging, and error-message tests (`TestMCPHealthCheckTool`). Full suite passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added authentication credential validation for MCP servers at startup via health-check tools * Health-check tools are auto-detected from identity tools or can be explicitly configured * Improved error messages when authentication validation fails * **Documentation** * New section documenting health-check validation, configuration, and troubleshooting * **Chores** * Updated default Helm values to include context tool for authentication validation  --------- Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: alonelish <alon.elish@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Roi Glinik <groi.tech@gmail.com>
diff --git a/docs/data-sources/remote-mcp-servers.md b/docs/data-sources/remote-mcp-servers.md
@@ -391,6 +391,32 @@ MCP servers can forward HTTP headers from the incoming request to the MCP backen
 
 For full details on template syntax, blocked headers, precedence rules, and examples for other toolset types, see [HTTP Header Propagation](header-propagation.md).
 
+**Validating Authentication at Startup (Health Check)**
+
+Many MCP servers return their full tool list even when the configured credential is invalid (a bad token, expired key, etc.). Because of this, simply connecting and listing tools does not prove that authentication works — the toolset can appear "connected" while every real call would fail.
+
+To catch this at startup, Holmes can invoke a single read-only tool during the toolset's health check. If the call fails (e.g. `401 Unauthorized`), the toolset is marked **disabled** with the underlying error surfaced, instead of showing as enabled.
+
+- **Automatic:** if the server exposes a well-known identity tool — `get_me`, `get_current_user`, `get_authenticated_user`, or `whoami` — Holmes auto-detects and uses it. No configuration needed.
+- **Explicit:** for any other server, set `health_check_tool` to a tool of your choice. This takes precedence over auto-detection.
+
+The chosen tool **must be read-only, take no required arguments** (it is called with empty arguments `{}`), and actually exercise the credential. If `health_check_tool` is unset and the server exposes none of the known identity tools, the auth health check is skipped and the toolset loads as long as listing tools succeeds.
+
+```yaml
+mcp_servers:
+  my_server:
+    description: "My custom MCP server"
+    config:
+      url: "http://my-mcp:8000/mcp"
+      mode: streamable-http
+      headers:
+        Authorization: "Bearer {{ env.MY_API_KEY }}"
+      # Invoked with empty args at startup to verify the credential is valid.
+      # Must be a read-only, no-argument tool exposed by your server.
+      health_check_tool: get_current_user
+    llm_instructions: "..."
+```
+
 ## Configuration Format Migration
 
 The MCP server configuration format has been updated. The `url` field must now be inside the `config` section.
diff --git a/helm/holmes/templates/toolset-config.yaml b/helm/holmes/templates/toolset-config.yaml
@@ -106,6 +106,7 @@ data:
         "url" (printf "http://%s-github-mcp-server.%s.svc.cluster.local:8000/mcp" .Release.Name .Release.Namespace)
         "mode" "streamable-http"
         "icon_url" "https://raw.githubusercontent.com/gilbarbara/logos/de2c1f96ff6e74ea7ea979b43202e8d4b863c655/logos/github.svg"
+        "health_check_tool" "get_me"
     }}
     {{- $githubMcpServers := dict "github" (dict
         "description" "GitHub MCP Server - access repositories, pull requests, issues, and GitHub Actions. Debug CI failures, search code, and delegate tasks to Copilot."
@@ -119,6 +120,7 @@ data:
         "url" (printf "http://%s-github-mcp-server.%s.svc.cluster.local:8000/sse" .Release.Name .Release.Namespace)
         "mode" "sse"
         "icon_url" "https://raw.githubusercontent.com/gilbarbara/logos/de2c1f96ff6e74ea7ea979b43202e8d4b863c655/logos/github.svg"
+        "health_check_tool" "get_me"
     }}
     {{- $githubMcpServers := dict "github" (dict
         "description" "GitHub MCP Server - access repositories, pull requests, issues, and GitHub Actions. Debug CI failures, search code, and delegate tasks to Copilot."
@@ -136,6 +138,7 @@ data:
           "url" (printf "http://%s-gitlab-mcp-server.%s.svc.cluster.local:8000/sse" .Release.Name .Release.Namespace)
           "mode" "sse"
           "icon_url" "https://raw.githubusercontent.com/gilbarbara/logos/de2c1f96ff6e74ea7ea979b43202e8d4b863c655/logos/gitlab.svg"
+          "health_check_tool" "get_current_user"
         )
         "llm_instructions" (include "holmes.gitlabMcp.llmInstructions" . | trim)
       )
diff --git a/helm/holmes/values.yaml b/helm/holmes/values.yaml
@@ -476,9 +476,12 @@ mcpAddons:
 
       # Toolsets to enable (comma-separated list of toolset names)
       # Available: repos, issues, pull_requests, actions, code_security, users, context, etc.
-      # Default: repos, issues, pull_requests, actions (covers most investigation needs)
+      # Default: repos, issues, pull_requests, actions, context (covers most investigation needs).
+      # 'context' exposes the read-only get_me tool, which Holmes uses to validate
+      # the GitHub token at startup (a bad PAT marks the toolset disabled instead of
+      # appearing connected). Keep 'context' enabled unless you have a reason not to.
       # Ignored when `tools` below is set (tools takes precedence as a hard allowlist).
-      toolsets: "repos,issues,pull_requests,actions"
+      toolsets: "repos,issues,pull_requests,actions,context"
 
       # Individual tools to enable (comma-separated, optional).
       # When set, restricts Holmes to EXACTLY this list of tools — takes precedence
diff --git a/holmes/plugins/toolsets/mcp/toolset_mcp.py b/holmes/plugins/toolsets/mcp/toolset_mcp.py
@@ -113,6 +113,20 @@ class MCPMode(str, Enum):
     STDIO = "stdio"
 
 
+# Well-known, read-only "who am I" tools used to verify MCP authentication when
+# no health_check_tool is explicitly configured. MCP servers commonly expose an
+# authenticated-identity endpoint (e.g. GitHub's get_me, GitLab's
+# get_current_user). Calling one with empty arguments is a side-effect-free way
+# to confirm credentials (such as an API token) are actually valid, since
+# list_tools succeeds even with a bad token. Order reflects matching priority.
+DEFAULT_HEALTH_CHECK_TOOLS: List[str] = [
+    "get_me",
+    "get_current_user",
+    "get_authenticated_user",
+    "whoami",
+]
+
+
 
 
 
@@ -164,6 +178,14 @@ class MCPConfig(ToolsetConfig):
         title="OAuth",
         description="OAuth authorization_code configuration. When set, users authenticate via browser before tools can be used.",
     )
+    health_check_tool: Optional[str] = Field(
+        default=None,
+        title="Health Check Tool",
+        description="Name of a read-only tool to invoke during health check to verify authentication works. "
+        "If set, this tool will be called with empty arguments after loading tools to ensure the "
+        "connection is fully functional (e.g., API token is valid). Example: 'get_me' for GitHub MCP.",
+        examples=["get_me", "get_current_user"],
+    )
 
     def get_lock_string(self) -> str:
         return str(self.url)
@@ -201,6 +223,14 @@ class StdioMCPConfig(ToolsetConfig):
         description="Icon URL for this MCP server, displayed in the UI for tool calls.",
         examples=["https://cdn.simpleicons.org/github/181717"],
     )
+    health_check_tool: Optional[str] = Field(
+        default=None,
+        title="Health Check Tool",
+        description="Name of a read-only tool to invoke during health check to verify authentication works. "
+        "If set, this tool will be called with empty arguments after loading tools to ensure the "
+        "connection is fully functional (e.g., API token is valid). Example: 'get_me' for GitHub MCP.",
+        examples=["get_me", "get_current_user"],
+    )
 
     def get_lock_string(self) -> str:
         return str(self.command)
@@ -932,6 +962,20 @@ def prerequisites_callable(self, config) -> Tuple[bool, str]:
             if not self.tools:
                 logging.warning("mcp server %s loaded 0 tools.", self.name)
 
+            # Invoke a read-only tool to verify authentication actually works.
+            # MCP servers (e.g. GitHub) often return their tool list even with
+            # invalid credentials, so listing tools alone doesn't prove auth.
+            # Use the explicitly-configured tool if set, otherwise auto-detect a
+            # well-known read-only identity tool from the loaded tools.
+            health_check_tool_name = (
+                self._mcp_config.health_check_tool
+                or self._auto_detect_health_check_tool()
+            )
+            if health_check_tool_name:
+                health_check_result = self._run_health_check_tool(health_check_tool_name)
+                if not health_check_result[0]:
+                    return health_check_result
+
             return (True, "")
         except Exception as e:
             error_detail = _extract_root_error_message(e)
@@ -941,6 +985,86 @@ def prerequisites_callable(self, config) -> Tuple[bool, str]:
                 ". If the server is still starting up, Holmes will retry automatically",
             )
 
+    def _auto_detect_health_check_tool(self) -> Optional[str]:
+        """Pick a default health-check tool from a known allowlist of read-only
+        identity tools when none is explicitly configured.
+
+        Returns the name of the first allowlisted tool the server exposes, or
+        None if it exposes none (in which case the auth health check is skipped).
+        """
+        tool_names = {t.name for t in self.tools}
+        for candidate in DEFAULT_HEALTH_CHECK_TOOLS:
+            if candidate in tool_names:
+                logging.info(
+                    "MCP server %s: no health_check_tool configured, auto-detected '%s' for auth validation",
+                    self.name,
+                    candidate,
+                )
+                return candidate
+        # No identity tool exposed — the auth health check is skipped. Log it so
+        # a silently-skipped check is diagnosable (e.g. GitHub MCP without its
+        # 'context' toolset enabled does not expose get_me). Set an explicit
+        # health_check_tool in config to validate auth via a different tool.
+        logging.debug(
+            "MCP server %s: no health_check_tool configured and none of %s are "
+            "exposed by the server; skipping auth health check",
+            self.name,
+            DEFAULT_HEALTH_CHECK_TOOLS,
+        )
+        return None
+
+    def _run_health_check_tool(self, tool_name: str) -> Tuple[bool, str]:
+        """Invoke a tool to verify authentication actually works.
+
+        MCP servers may return a tool list even with invalid credentials (e.g., bad
+        GitHub token). This method calls a specified read-only tool with empty
+        arguments to verify the connection is fully functional.
+        """
+        matching_tools = [t for t in self.tools if t.name == tool_name]
+        if not matching_tools:
+            available = [t.name for t in self.tools]
+            return (
+                False,
+                f"MCP server {self.name}: health_check_tool '{tool_name}' not found. "
+                f"Available tools: {', '.join(available[:10])}{'...' if len(available) > 10 else ''}",
+            )
+
+        try:
+            result = asyncio.run(self._call_health_check_tool_async(tool_name))
+            if result.isError:
+                error_chunks = [
+                    RemoteMCPTool._extract_text_from_content_block(c)
+                    for c in result.content
+                ]
+                error_text = " ".join(t for t in error_chunks if t)
+                logging.warning(
+                    "MCP server %s health check failed (tool: %s): %s",
+                    self.name, tool_name, error_text or "unknown error"
+                )
+                return (
+                    False,
+                    f"MCP server {self.name}: health check tool '{tool_name}' with params {{}} "
+                    f"failed - {error_text or 'unknown error'}",
+                )
+            logging.info("MCP server %s health check passed (tool: %s)", self.name, tool_name)
+            return (True, "")
+        except Exception as e:
+            error_detail = _extract_root_error_message(e)
+            logging.warning(
+                "MCP server %s health check exception (tool: %s): %s",
+                self.name, tool_name, error_detail
+            )
+            return (
+                False,
+                f"MCP server {self.name}: health check tool '{tool_name}' with params {{}} "
+                f"failed - {error_detail}",
+            )
+
+    async def _call_health_check_tool_async(self, tool_name: str):
+        """Call a tool during health check (no request context available)."""
+        async with get_initialized_mcp_session(self, None) as session:
+            return await session.call_tool(tool_name, {})
+
     def _check_oauth_server_reachable(self) -> Tuple[bool, str]:
         """For OAuth MCP servers, verify reachability without authenticating.
 
diff --git a/tests/test_mcp_toolset.py b/tests/test_mcp_toolset.py
