Doc 1846: Remote MCP GA (#472)

* Remove beta labels, single-source duplicated content, and add testing

* Document automatic service account creation

* Add billing info for MCP

* Correct MCP and serverless cluster API paths

- Change MCP API paths from /v1/mcp_servers to /v1/redpanda-connect/mcp-servers
- Fix serverless cluster path from /v1/serverless-clusters to /v1/serverless/clusters
- Add note about Data Plane API URL availability timing (5-10 minute delay)
- Add required service_account field to MCP server creation example

* Split admin guide into task-based pages

- Restructure admin-guide.adoc as navigation hub
- Create manage-servers.adoc for lifecycle management (edit, stop, start, delete)
- Create scale-resources.adoc for resource allocation and cost optimization
- Create monitor-activity.adoc for OpenTelemetry traces and observability
- Add Grafana Cloud integration example to monitoring documentation
- Update navigation to show hierarchical structure with three child pages

This improves information architecture and user experience by organizing
content by specific tasks rather than mixing multiple topics in one page.

* Enhance MCP server pipeline patterns documentation

- Add pattern selection guide at top for quick navigation
- Add 'When to use' and 'Example use cases' to all patterns
- Add new pattern sections:
  * Data generators (generate input for testing)
  * Data consumers (reading from Redpanda topics)
  * Data transformation (Bloblang mapping examples)
  * AI/LLM integration (OpenAI chat completion and embeddings)
- Fix deprecated kafka_franz component, use redpanda input instead
- Remove unnecessary manual anchor IDs (Antora generates them automatically)
- Restructure Redpanda integration section with proper hierarchy

These improvements make the patterns catalog more discoverable and
provide clearer guidance on when to use each pattern.

* Add .env to gitignore

Prevent environment files from being committed to repository.

* Escape curly braces in API path parameters

AsciiDoc formatting: escape curly braces in inline code blocks
to prevent them from being interpreted as attribute references.

* Fix anchor

* Document tags and reserved tag names for Remote MCP servers

* Update MCP server creation flow to match UI terminology

* Convert tagging strategies to table format

* Update workflow and document inline secret creation in Cloud Console

* Add cross-reference to tool contract design section

* Simplify heading to 'Build your MCP tool'

* Add real trace examples showing tool execution and error events

* Use uppercase snakecase for Grafana Cloud secret names

* Simplify monitoring docs and remove unnecessary sections

* Update monitoring docs with realistic trace examples and schema details

- Change heading to 'OpenTelemetry traces topic' for clarity
- Document auto-created Protobuf schema 'redpanda.otel_traces-value'
- Update trace data structure example with realistic values (hex IDs, numeric enums)
- Refine field descriptions and reference schema for details
- Remove Custom dashboards and Best practices sections (too implementation-specific)

* Update branch

* Fix doc-tools not found error in test-mcp-examples workflow

- Add npm install step to install @redpanda-data/docs-extensions-and-macros
- Use --no-install flag with npx to prevent trying to download from npm
- The doc-tools binary is provided by @redpanda-data/docs-extensions-and-macros

* Make MCP validation failures fail the test suite

- Add MCP_FAILS counter to track validation failures
- Remove '|| true' that was suppressing validation failures
- Increment MCP_FAILS when validate_mcp_metadata returns non-zero
- Display MCP validation failures in test summary
- Exit with non-zero code when MCP validations fail

This ensures missing descriptions or other MCP metadata issues cause
the GitHub Actions workflow to fail.

* fix: Resolve ShellCheck warnings in test script

Separate variable declaration from assignment in validate_mcp_metadata
to avoid masking return values and resolve ShellCheck warnings.

* Fix schema syntax

* Remove beta

* Add badge macro

* Add badge macro

* fix: Correct MCP API examples and secret naming

- Fix update_mask to be query parameter in scale-resources.adoc
- Fix SASL configuration to use array format in pipeline-patterns.adoc
- Update secret references to use uppercase snakecase (OPENAI_API_KEY)
- All changes follow OpenAPI spec and documented conventions

* Add warning about redpanda.otel_traces topic management

   - Document automatic recreation of topic and schema if deleted
   - Warn that deleting topic permanently deletes all trace data
   - Clarify that users should not produce to this system-managed topic

* Document redpanda.otel_traces topic billing

   - Clarify that Serverless clusters are not billed for traces topic usage
   - Note that Dedicated/BYOC clusters follow standard billing for traces

* Add 'Outputs with processors' pattern for Remote MCP

- Create reusable example showing output with LLM processor
- Address customer confusion about combining processors with outputs
- Clarify that outputs CAN have processors, but processor tools CANNOT have outputs

* Clarify how to identify MCP tools in shared traces topic

- Explain that all MCP servers write to the same redpanda.otel_traces topic
- Document how to identify which tool by instrumentationScope and name fields
- Clarify rpcn-mcp vs benthos naming for parent and child spans
- Help customers filter traces in multi-server environments

* Add trace topic configuration and lifecycle details

* Minor fixes

* Use 'MCP server' terminology consistently across MCP remote docs

* Fix remaining 'Remote MCP server' instances to use 'MCP server'

* Remove 'Remote MCP' from xref link text and section headings

* Clarify terms

* Apply suggestions from code review

Co-authored-by: Michele Cyran <michele@redpanda.com>

* Formatting

* Fix tests to use mcp-server lint

* Cleanup

* Use verbose linting and check for output since Connect does not exit with error code 1

* Fix linter

* Fix MCP trace docs: clarify 100% sampling, remove unsupported exports

  Remove Grafana and Datadog export instructions since one-span-per-topic
  format is incompatible with batch-based platform requirements.

* Update monitor-activity.adoc

* Update modules/ai-agents/pages/mcp/remote/monitor-activity.adoc

* Update developer-guide.adoc

* apply suggestions

* apply suggestions

---------

Co-authored-by: Michele Cyran <michele@redpanda.com>

Author: JakeSCahill

.github/workflows/test-mcp-examples.yaml

@@ -0,0 +1,53 @@
+name: Test MCP Examples
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'modules/ai-agents/examples/**/*.yaml'
+      - 'modules/ai-agents/examples/test-mcp-examples.sh'
+  pull_request:
+    branches:
+      - main
+    paths:
+      - 'modules/ai-agents/examples/**/*.yaml'
+      - 'modules/ai-agents/examples/test-mcp-examples.sh'
+
+jobs:
+  test-all-examples:
+    name: Test All MCP Examples
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install npm dependencies
+        run: npm install
+
+      - name: Install doc-tools dependencies
+        run: npx --no-install doc-tools install-test-dependencies
+
+      - name: Install yq
+        run: |
+          sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
+          sudo chmod +x /usr/local/bin/yq
+
+      - name: Make test script executable
+        run: chmod +x modules/ai-agents/examples/test-mcp-examples.sh
+
+      - name: Run MCP examples tests
+        run: |
+          cd modules/ai-agents/examples
+          ./test-mcp-examples.sh
+
+      - name: Test Summary
+        if: always()
+        run: |
+          echo "::notice title=MCP Examples Testing::All Cloud MCP examples have been validated"

.github/workflows/update-extensions.yml

@@ -32,7 +32,7 @@ jobs:
 
       - uses: actions/setup-node@v4
         with:
-          node-version: '18'
+          node-version: '22'
 
       - run: npm install
 

.gitignore

@@ -8,3 +8,4 @@ build/_redirects
 poc-docs/
 docs/
 .docusaurus
+.env

local-antora-playbook.yml

@@ -37,6 +37,7 @@ asciidoc:
   - '@redpanda-data/docs-extensions-and-macros/macros/config-ref'
   - '@redpanda-data/docs-extensions-and-macros/macros/helm-ref'
   - '@redpanda-data/docs-extensions-and-macros/asciidoc-extensions/add-line-numbers-highlights'
+  - '@redpanda-data/docs-extensions-and-macros/macros/badge'
 antora:
   extensions:
   - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-info'

modules/ROOT/nav.adoc

@@ -80,6 +80,9 @@
 *** xref:ai-agents:mcp/remote/quickstart.adoc[Quickstart]
 *** xref:ai-agents:mcp/remote/developer-guide.adoc[Developer Guide]
 *** xref:ai-agents:mcp/remote/admin-guide.adoc[Admin Guide]
+**** xref:ai-agents:mcp/remote/manage-servers.adoc[Manage Servers]
+**** xref:ai-agents:mcp/remote/scale-resources.adoc[Scale Resources]
+**** xref:ai-agents:mcp/remote/monitor-activity.adoc[Monitor Activity]
 *** xref:ai-agents:mcp/remote/pipeline-patterns.adoc[MCP Server Patterns]
 
 * xref:develop:connect/about.adoc[Redpanda Connect]

modules/ai-agents/examples/redpanda_output_with_processors.yaml

@@ -0,0 +1,19 @@
+redpanda:
+  seed_brokers: [ "${REDPANDA_BROKERS}" ]
+  topic: "llm-responses"
+  tls:
+    enabled: true
+  sasl:
+    - mechanism: "${REDPANDA_SASL_MECHANISM}"
+      username: "${REDPANDA_SASL_USERNAME}"
+      password: "${REDPANDA_SASL_PASSWORD}"
+
+processors:
+  - openai_chat_completion:
+      api_key: "${secrets.OPENAI_API_KEY}"
+      model: "gpt-4"
+      prompt: ${! json("question") }
+  - mapping: |
+      root.question = this.question
+      root.answer = this.content
+      root.timestamp = now().ts_format("2006-01-02T15:04:05Z07:00")

modules/ai-agents/examples/test-mcp-examples.sh

@@ -0,0 +1,157 @@
+#!/usr/bin/env bash
+#
+# Automated testing script for Redpanda Connect MCP examples
+#
+# Usage:
+#   ./test-mcp-examples.sh
+
+set -euo pipefail
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+# Counters
+TOTAL=0
+SKIPPED=0
+MCP_FAILS=0
+
+echo "🧪 Redpanda Connect MCP Examples Test Suite"
+echo "============================================"
+echo ""
+
+# Run MCP server lint on the directory
+echo "Running rpk connect mcp-server lint..."
+LINT_OUTPUT=$(rpk connect mcp-server lint --skip-env-var-check --verbose 2>&1) || {
+    echo -e "${RED}❌ Linting failed${NC}"
+    echo ""
+    echo "$LINT_OUTPUT"
+    exit 1
+}
+echo -e "${GREEN}✅ Linting passed${NC}"
+echo ""
+
+# Function to validate MCP metadata
+validate_mcp_metadata() {
+    local file=$1
+
+    echo -n "  Validating MCP metadata... "
+
+    # Determine which YAML parser to use
+    local use_yq=true
+    if ! command -v yq &> /dev/null; then
+        use_yq=false
+        if ! command -v python3 &> /dev/null; then
+            echo -e "${RED}FAILED${NC} (neither yq nor python3 available)"
+            MCP_FAILS=$((MCP_FAILS + 1))
+            return 1
+        fi
+    fi
+
+    # Check if .meta.mcp exists
+    local mcp_exists
+    if $use_yq; then
+        mcp_exists=$(yq eval '.meta.mcp' "$file" 2>/dev/null)
+    else
+        mcp_exists=$(python3 -c "
+import yaml
+try:
+    with open('$file') as f:
+        doc = yaml.safe_load(f)
+    meta = doc.get('meta', {}) if doc else {}
+    mcp = meta.get('mcp')
+    print('null' if mcp is None else 'exists')
+except:
+    print('null')
+" 2>/dev/null)
+    fi
+
+    if [[ "$mcp_exists" == "null" || -z "$mcp_exists" ]]; then
+        echo -e "${YELLOW}SKIPPED${NC} (no MCP metadata)"
+        SKIPPED=$((SKIPPED + 1))
+        return 0
+    fi
+
+    # Read .meta.mcp.enabled
+    local enabled
+    if $use_yq; then
+        enabled=$(yq eval '.meta.mcp.enabled' "$file" 2>/dev/null)
+    else
+        enabled=$(python3 -c "
+import yaml
+try:
+    with open('$file') as f:
+        doc = yaml.safe_load(f)
+    enabled = doc.get('meta', {}).get('mcp', {}).get('enabled')
+    print('null' if enabled is None else str(enabled).lower())
+except:
+    print('null')
+" 2>/dev/null)
+    fi
+
+    if [[ "$enabled" != "true" ]]; then
+        echo -e "${YELLOW}WARNING${NC} (mcp.enabled not set to true)"
+        return 0
+    fi
+
+    # Read .meta.mcp.description
+    local description
+    if $use_yq; then
+        description=$(yq eval '.meta.mcp.description' "$file" 2>/dev/null)
+    else
+        description=$(python3 -c "
+import yaml
+try:
+    with open('$file') as f:
+        doc = yaml.safe_load(f)
+    desc = doc.get('meta', {}).get('mcp', {}).get('description')
+    print('null' if desc is None or desc == '' else str(desc))
+except:
+    print('null')
+" 2>/dev/null)
+    fi
+
+    if [[ "$description" == "null" || -z "$description" ]]; then
+        echo -e "${RED}FAILED${NC} (missing description)"
+        MCP_FAILS=$((MCP_FAILS + 1))
+        return 1
+    fi
+
+    echo -e "${GREEN}PASSED${NC}"
+    return 0
+}
+
+# Validate MCP metadata for each file
+for file in *.yaml; do
+    if [[ -f "$file" ]]; then
+        TOTAL=$((TOTAL + 1))
+        echo ""
+        echo -e "${BLUE}📄 Validating: $file${NC}"
+        validate_mcp_metadata "$file"
+    fi
+done
+
+# Summary
+echo ""
+echo "============================================"
+echo "📊 Test Summary"
+echo "============================================"
+echo "Total configs tested: $TOTAL"
+if [[ $MCP_FAILS -gt 0 ]]; then
+    echo -e "MCP validation failures: ${RED}$MCP_FAILS${NC}"
+fi
+if [[ $SKIPPED -gt 0 ]]; then
+    echo -e "Skipped: ${YELLOW}$SKIPPED${NC}"
+fi
+echo ""
+
+if [[ $MCP_FAILS -gt 0 ]]; then
+    echo -e "${RED}❌ Some tests failed${NC}"
+    exit 1
+else
+    echo -e "${GREEN}✅ All tests passed!${NC}"
+    exit 0
+fi