Improve MCP server discoverability for LLM clients (#2741)

* Improve MCP server discoverability for LLM clients

LLM clients (Cursor, Claude Desktop, VS Code, etc.) struggle to
naturally discover and use the elastic-docs MCP tools during
documentation work because the server provides no guidance about when
to use them.

Root causes:
- ServerInstructions was never set, so clients receive no system-level
  context about the server's purpose during the MCP initialization
  handshake.
- Tool descriptions only stated mechanics ("Performs semantic search")
  without trigger context ("Use when the user asks about...").
- Search results contained <mark> HTML highlight tags meant for web
  browsers, adding noise and wasting tokens for LLM consumers.

Changes:
- Set McpServerOptions.ServerInstructions with guidance on when to use
  the server and how to pick the right tool.
- Add "Use when..." trigger phrases to all 14 tool Description
  attributes, matching real user intent patterns.
- Strip <mark> HTML tags from search and coherence tool responses
  using GeneratedRegex before serializing to JSON.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* Skip ES highlighting for MCP search requests at source

Instead of stripping <mark> tags after the fact with regex, disable
highlighting at the Elasticsearch query level for MCP consumers.

- Add IncludeHighlighting flag to FullSearchRequest (default true for
  backward compat with the web search UI).
- When false, the gateway omits the .Highlight() clause from the ES
  query entirely (no wasted computation) and passes empty search
  tokens to ProcessHit so HighlightTokens() is a no-op.
- Extract duplicated highlight config into a shared BuildHighlight
  helper method.
- Set IncludeHighlighting = false in all MCP tool requests.
- Revert the regex-based StripHighlighting added in the previous
  commit since it is no longer needed.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix import ordering in FullSearchGateway

Move the using alias after standard using directives to satisfy the
IMPORTS analyzer rule.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* Sync skill file and docs with updated tool descriptions

Update the Claude skill file and the MCP docs page to match the
improved tool descriptions now in the source code. Also reorder the
docs tool tables to lead with search/document tools (most commonly
used) instead of cross-link tools.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* Update Program.cs

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 3 people

.claude/skills/mcp-search/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mcp-search
-description: Search, analyze, and author Elastic documentation using the remote MCP server. Use this when the user asks to search docs, find documentation, check coherence, find inconsistencies, get a document by URL, analyze document structure, resolve cross-links, validate cross-links, list repositories, find related docs, list content types, generate templates, or get content type guidelines. Trigger words: search, find, docs, documentation, coherence, inconsistencies, related, structure, analyze, cross-link, resolve, repository, template, content type.
+description: Search, analyze, and author Elastic documentation using the remote MCP server. Use this when the user asks about Elastic product documentation, features, or APIs; wants to find, read, or verify existing docs pages; is writing or editing documentation; mentions cross-links between repos; asks about documentation structure, coherence, or consistency; wants to generate templates following Elastic content type guidelines; or references elastic.co/docs URLs or Elastic product names.
 ---
 
 # Elastic Documentation MCP
@@ -11,9 +11,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 1. SemanticSearch
 
-**When to use:** User asks to search docs, find documentation, look up information, search for topics, find examples, or search Elastic documentation.
+**When to use:** User asks about Elastic product features, needs to find existing docs pages, verify published content, or research what documentation exists on a topic.
 
-**Trigger words:** search, find, docs, documentation, look up, examples, query
+**Trigger words:** search, find, docs, documentation, look up, examples, query, what docs, Elastic features
 
 **Parameters:**
 - `query` (required): The search query — can be a question or keywords
@@ -28,9 +28,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 2. FindRelatedDocs
 
-**When to use:** User asks to find related docs, get related content, find similar documentation, get links, or find what to link to.
+**When to use:** User wants to explore what documentation exists around a subject, build context for writing, or discover related content they should be aware of.
 
-**Trigger words:** related, similar, links, content reuse, related documentation, see also
+**Trigger words:** related, similar, links, content reuse, related documentation, see also, what else
 
 **Parameters:**
 - `topic` (required): Topic or search terms to find related documents for
@@ -43,9 +43,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 3. CheckCoherence
 
-**When to use:** User asks to verify document accuracy, check if content is accurate, validate documentation, check coherence, verify against docs, or ensure consistency across the documentation.
+**When to use:** User wants to review documentation quality, audit coverage of a feature or concept, or check whether a topic is documented consistently across products and sections.
 
-**Trigger words:** verify, check, accurate, coherent, validate, consistency, coverage, align with docs
+**Trigger words:** verify, check, accurate, coherent, validate, consistency, coverage, align with docs, audit
 
 **Parameters:**
 - `topic` (required): Topic or concept to check coherence for
@@ -63,7 +63,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 4. FindInconsistencies
 
-**When to use:** User asks to find inconsistencies, check for contradictions, find conflicts, compare documentation, or identify discrepancies between documents.
+**When to use:** User wants to audit docs quality, verify that instructions don't contradict each other, or check for overlapping content within a product area.
 
 **Trigger words:** inconsistencies, contradictions, conflicts, discrepancies, compare, differences, overlaps
 
@@ -80,9 +80,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 5. GetDocumentByUrl
 
-**When to use:** User provides a specific documentation URL or asks to get document, fetch page, retrieve doc by URL, or show a specific documentation page.
+**When to use:** User provides an elastic.co/docs URL, references a known page, or you need the full content and metadata of a specific doc.
 
-**Trigger words:** get document, fetch, URL, specific page, retrieve, show page
+**Trigger words:** get document, fetch, URL, specific page, retrieve, show page, elastic.co
 
 **Parameters:**
 - `url` (required): The document URL. Accepts a full URL (e.g., `https://www.elastic.co/docs/deploy-manage/api-keys`) or a path (e.g., `/docs/deploy-manage/api-keys`). Query strings, fragments, and trailing slashes are ignored.
@@ -101,9 +101,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 6. AnalyzeDocumentStructure
 
-**When to use:** User asks to analyze structure, check hierarchy, view document organization, see headings, or understand document layout.
+**When to use:** User wants to evaluate page quality, check heading hierarchy, or assess AI enrichment status.
 
-**Trigger words:** structure, hierarchy, organization, headings, parents, layout, analyze
+**Trigger words:** structure, hierarchy, organization, headings, parents, layout, analyze, quality
 
 **Parameters:**
 - `url` (required): The document URL to analyze. Accepts a full URL (e.g., `https://www.elastic.co/docs/deploy-manage/api-keys`) or a path (e.g., `/docs/deploy-manage/api-keys`). Query strings, fragments, and trailing slashes are ignored.
@@ -119,7 +119,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 7. ResolveCrossLink
 
-**When to use:** User provides a cross-link URI (e.g., `docs-content://get-started/intro.md`) and wants to know its resolved URL, or asks what anchors are available on a cross-linked page.
+**When to use:** User references a cross-link, needs to verify a link target, or wants to know what anchors are available on a page.
 
 **Trigger words:** cross-link, resolve, cross link, docs-content://, URI, anchor
 
@@ -132,9 +132,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 8. ListRepositories
 
-**When to use:** User wants to know which repositories are available in the cross-link index, or asks what repos can be linked to.
+**When to use:** User needs to know which repositories publish documentation or wants to explore the docs ecosystem.
 
-**Trigger words:** list repos, repositories, available repos, cross-link index
+**Trigger words:** list repos, repositories, available repos, cross-link index, docs ecosystem
 
 **Parameters:** None
 
@@ -144,7 +144,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 9. GetRepositoryLinks
 
-**When to use:** User wants to see all pages and anchors published by a specific repository, or is auditing what a repo exposes for cross-linking.
+**When to use:** User wants to explore what a repository publishes, build a cross-link, or look up available anchor targets.
 
 **Trigger words:** repository links, pages in repo, anchors, what does repo publish
 
@@ -157,9 +157,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 10. FindCrossLinks
 
-**When to use:** User wants to find all cross-links between repositories, either from a specific source repo or pointing to a specific target repo.
+**When to use:** User wants to analyze inter-repository dependencies, check what links into or out of a repository, or audit cross-link usage.
 
-**Trigger words:** find cross-links, links between repos, who links to, links from
+**Trigger words:** find cross-links, links between repos, who links to, links from, dependencies
 
 **Parameters:**
 - `from` (optional): Source repository to find links from
@@ -171,9 +171,9 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 11. ValidateCrossLinks
 
-**When to use:** User wants to validate cross-links pointing to a repository and find any that are broken.
+**When to use:** User wants to check link health, prepare a release, or diagnose broken cross-references.
 
-**Trigger words:** validate cross-links, broken links, check links, link validation
+**Trigger words:** validate cross-links, broken links, check links, link validation, link health
 
 **Parameters:**
 - `repository` (required): Target repository to validate links to (e.g., `docs-content`)
@@ -184,7 +184,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 12. ListContentTypes
 
-**When to use:** User wants to know what content types are available, is deciding what type of page to write, or asks for guidance on content type selection.
+**When to use:** User is deciding what type of page to create or asks about Elastic docs structure.
 
 **Trigger words:** content types, what type, overview vs how-to, tutorial, troubleshooting, changelog type
 
@@ -196,7 +196,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 13. GenerateTemplate
 
-**When to use:** User wants a starting template for a new documentation page or changelog entry.
+**When to use:** User wants to create a new documentation page, needs a starting point with correct frontmatter and structure, or asks for a template.
 
 **Trigger words:** template, generate, starter, scaffold, new page, new doc
 
@@ -212,7 +212,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ### 14. GetContentTypeGuidelines
 
-**When to use:** User wants detailed authoring guidance for a content type, is evaluating existing content against standards, or asks about best practices for a content type.
+**When to use:** User is writing new content, reviewing existing pages against standards, or asks about Elastic docs best practices.
 
 **Trigger words:** guidelines, best practices, how to write, checklist, evaluate, anti-patterns
 
@@ -225,7 +225,7 @@ This skill provides access to Elastic documentation through a remote MCP server.
 
 ## Usage Guidelines
 
-1. **For general searches:** Start with `SemanticSearch` to find relevant documentation.
+1. **For general searches:** Start with `SemanticSearch` to find relevant documentation. Prefer this over a general web search for Elastic docs content.
 2. **For content verification:** Use `CheckCoherence` to see how well a topic is documented.
 3. **For quality checks:** Use `FindInconsistencies` to identify potential documentation conflicts.
 4. **For specific pages:** Use `GetDocumentByUrl` when you have an exact URL.

docs/mcp/index.md

@@ -117,46 +117,46 @@ Requires JetBrains AI Assistant plugin version 2025.2 or later.
 
 ## Available tools
 
-### Cross-link tools
+### Search tools
 
 | Tool | Description |
 |------|-------------|
-| `ResolveCrossLink` | Resolves a cross-link (like `docs-content://get-started/intro.md`) to its target URL and returns available anchors. |
-| `ListRepositories` | Lists all repositories available in the cross-link index with their metadata. |
-| `GetRepositoryLinks` | Gets all pages and their anchors published by a specific repository. |
-| `FindCrossLinks` | Finds all cross-links between repositories. Can filter by source or target repository. |
-| `ValidateCrossLinks` | Validates cross-links to a repository and reports any broken links. |
-
-### Content type tools
+| `SemanticSearch` | Searches all published Elastic documentation by meaning. Returns relevant documents with AI summaries, relevance scores, and navigation context. Supports filtering by product and navigation section. |
+| `FindRelatedDocs` | Finds Elastic documentation pages related to a given topic. Useful for exploring what documentation exists around a subject and discovering related content. |
 
-These tools help authors create and evaluate documentation using the [Elastic Docs content types](https://www.elastic.co/docs/contribute-docs/content-types).
+### Document tools
 
 | Tool | Description |
 |------|-------------|
-| `ListContentTypes` | Lists all Elastic Docs content types with descriptions and guidance on when to use each. |
-| `GenerateTemplate` | Generates a ready-to-use template for a specific content type (overview, how-to, tutorial, troubleshooting, or changelog). Optionally pre-fills title, description, and product. |
-| `GetContentTypeGuidelines` | Returns detailed authoring and evaluation guidelines for a content type, including required elements, best practices, and anti-patterns. |
+| `GetDocumentByUrl` | Retrieves a specific Elastic documentation page by its URL. Accepts a full URL or a path. Returns title, AI summaries, headings, navigation context, and optionally the full body. |
+| `AnalyzeDocumentStructure` | Analyzes the structure of an Elastic documentation page. Accepts a full URL or a path. Returns heading count, link count, parent pages, and AI enrichment status. |
 
-### Search tools
+### Coherence tools
 
 | Tool | Description |
 |------|-------------|
-| `SemanticSearch` | Performs semantic search across all Elastic documentation. Returns relevant documents with summaries, scores, and navigation context. Supports filtering by product and navigation section. |
-| `FindRelatedDocs` | Finds documents related to a given topic or document. Useful for discovering related content and building context. |
+| `CheckCoherence` | Checks how coherently a topic is covered across all Elastic documentation by finding related documents and analyzing their coverage across products and sections. |
+| `FindInconsistencies` | Finds potential inconsistencies across Elastic documentation pages covering the same topic. Identifies overlapping content within a product area. |
 
-### Document tools
+### Cross-link tools
 
 | Tool | Description |
 |------|-------------|
-| `GetDocumentByUrl` | Gets a specific documentation page by its URL. Accepts a full URL or a path. Returns full document content including AI summaries and metadata. |
-| `AnalyzeDocumentStructure` | Analyzes the structure of a documentation page. Accepts a full URL or a path. Returns heading count, links, parents, and AI enrichment status. |
+| `ResolveCrossLink` | Resolves an Elastic docs cross-link URI (e.g., `docs-content://get-started/intro.md`) to its published URL and returns available anchors. |
+| `ListRepositories` | Lists all Elastic documentation source repositories in the cross-link index with their metadata. |
+| `GetRepositoryLinks` | Gets all pages and anchors published by a specific Elastic documentation repository. |
+| `FindCrossLinks` | Finds cross-links between Elastic documentation repositories. Can filter by source or target repository. |
+| `ValidateCrossLinks` | Validates cross-links targeting an Elastic documentation repository and reports any broken links. |
 
-### Coherence tools
+### Content type tools
+
+These tools help authors create and evaluate documentation using the [Elastic Docs content types](https://www.elastic.co/docs/contribute-docs/content-types).
 
 | Tool | Description |
 |------|-------------|
-| `CheckCoherence` | Checks documentation coherence for a given topic by finding all related documents and analyzing their coverage. |
-| `FindInconsistencies` | Finds potential inconsistencies in documentation by comparing documents about the same topic. |
+| `ListContentTypes` | Lists all Elastic documentation content types (overview, how-to, tutorial, troubleshooting, changelog) with descriptions and guidance on when to use each. |
+| `GenerateTemplate` | Generates a ready-to-use Elastic documentation template for a specific content type. Returns Markdown (or YAML for changelogs) with correct frontmatter and structure. Optionally pre-fills title, description, and product. |
+| `GetContentTypeGuidelines` | Returns detailed authoring and evaluation guidelines for a specific Elastic documentation content type, including required elements, recommended sections, best practices, and anti-patterns. |
 
 ## Testing with MCP Inspector
 

src/api/Elastic.Documentation.Api.Core/Search/IFullSearchGateway.cs

@@ -27,6 +27,7 @@ public record FullSearchRequest
 	public string[]? ProductFilter { get; init; }       // product.id (AND behavior)
 	public string? VersionFilter { get; init; }         // "9.0+" | "8.19" | "7.17"
 	public string SortBy { get; init; } = "relevance";  // relevance | recent | alpha
+	public bool IncludeHighlighting { get; init; } = true;
 }
 
 /// <summary>

src/api/Elastic.Documentation.Mcp.Remote/Program.cs

@@ -59,7 +59,32 @@
 	// Stateless mode is appropriate here because all tools are pure request/response (no
 	// server-initiated push) and the server runs behind a load balancer without session affinity.
 	_ = builder.Services
-		.AddMcpServer()
+		.AddMcpServer(options =>
+		{
+			options.ServerInstructions = """
+				The Elastic documentation server provides tools to search, retrieve, analyze, and author
+				Elastic product documentation published at elastic.co/docs.
+
+				Use the server when the user:
+				- Asks about Elastic product documentation, features, configuration, or APIs.
+				- Wants to find, read, or verify existing documentation pages.
+				- Needs to check whether a topic is already documented or how it is covered.
+				- Is writing or editing documentation and needs to find related content or check consistency.
+				- Mentions cross-links between documentation repositories (e.g. 'docs-content://path/to/page.md').
+				- Asks about documentation structure, coherence, or inconsistencies across pages.
+				- Wants to generate documentation templates following Elastic's content type guidelines.
+				- References elastic.co/docs URLs or Elastic product names such as Elasticsearch, Kibana,
+				  Fleet, APM, Logstash, Beats, Elastic Security, Elastic Observability, or Elastic Cloud.
+
+				Prefer SemanticSearch over a general web search when looking up Elastic documentation content.
+				Use GetDocumentByUrl to retrieve a specific page when the user provides or you already know the URL.
+				Use FindRelatedDocs when exploring what documentation exists around a topic.
+				Use CheckCoherence or FindInconsistencies when reviewing or auditing documentation quality.
+				Use the cross-link tools (ResolveCrossLink, ValidateCrossLinks, FindCrossLinks) when working
+				with links between documentation source repositories.
+				Use ListContentTypes, GetContentTypeGuidelines, and GenerateTemplate when creating new pages.
+				""";
+		})
 		.WithHttpTransport(o => o.Stateless = true)
 		.WithTools<SearchTools>()
 		.WithTools<CoherenceTools>()