Skip to content

feat(mcp): Simplify deployment of Replication MCP server over HTTPS #1069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Ella6882 wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(mcp): Simplify deployment of Replication MCP server over HTTPS #1069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions airbyte/mcp/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,29 @@
2. Register the MCP server with your MCP client.
3. Test the MCP server connection using your MCP client.

### Hosted HTTPS deployment (remote MCP clients)

Remote MCP clients require a public **HTTPS** endpoint. They cannot use the local
stdio transport.

Use the **`airbyte-mcp-http`** entry point instead, which serves Streamable HTTP on port
8080. Terminate TLS at a reverse proxy or load balancer in front of the server.

```bash
# Docker (production transport)
docker build -f Dockerfile.mcp -t airbyte-mcp .
docker run --rm -p 8080:8080 \
-e AIRBYTE_MCP_ENV_FILE=/secrets/airbyte_mcp.env \
-v "/path/to/airbyte_mcp.env:/secrets/airbyte_mcp.env:ro" \
airbyte-mcp

# Local dev with the same transport as production
poe mcp-serve-streamable-http
```

For a full deployment guide (HTTPS, remote client integration, OIDC, CORS), see
[docs/MCP_HTTP_DEPLOYMENT.md](https://github.com/airbytehq/PyAirbyte/blob/main/docs/MCP_HTTP_DEPLOYMENT.md).

### Step 1: Generate a Dotenv Secrets File

To get started with the Airbyte Replication MCP server, you will need to create a dotenv
Expand Down
57 changes: 57 additions & 0 deletions airbyte/mcp/_http_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
"""HTTP transport configuration helpers for hosted MCP deployment."""

from __future__ import annotations

import os

from starlette.middleware import Middleware
from starlette.middleware.cors import CORSMiddleware


MCP_CORS_ORIGINS_ENV = "MCP_CORS_ORIGINS"

# Headers used by remote MCP clients over streamable HTTP.
_MCP_CLIENT_HEADERS = [
"mcp-protocol-version",
"mcp-session-id",
"Authorization",
"Content-Type",
"X-Airbyte-Workspace-Id",
"X-Airbyte-Cloud-Client-Id",
"X-Airbyte-Cloud-Client-Secret",
"X-Airbyte-Cloud-Api-Url",
"X-Airbyte-Cloud-Config-Api-Url",
]


def parse_cors_origins(raw: str | None) -> list[str]:
"""Parse a comma-separated list of CORS origins."""
if not raw:
return []
return [origin.strip() for origin in raw.split(",") if origin.strip()]


def build_http_middleware(
*,
cors_origins: str | None = None,
) -> list[Middleware]:
"""Build ASGI middleware for the hosted MCP HTTP transport.

When ``MCP_CORS_ORIGINS`` is set (comma-separated), enables CORS for
remote MCP clients that connect from a browser context (for example,
browser-based OAuth flows).
"""
origins = parse_cors_origins(cors_origins or os.getenv(MCP_CORS_ORIGINS_ENV, ""))

@coderabbitai coderabbitai Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Falsy "" falls through to env var — intentional?

cors_origins or os.getenv(...) treats an explicitly passed empty string the same as "unset", so it silently falls back to MCP_CORS_ORIGINS from the environment. That also makes test_build_http_middleware_without_origins (passing cors_origins="") implicitly depend on the env var being unset in CI. Want to switch to a None sentinel so callers can explicitly opt out of CORS regardless of env state?

💡 Proposed fix 
 def build_http_middleware(
     *,
     cors_origins: str | None = None,
 ) -> list[Middleware]:
     ...
-    origins = parse_cors_origins(cors_origins or os.getenv(MCP_CORS_ORIGINS_ENV, ""))
+    if cors_origins is None:
+        cors_origins = os.getenv(MCP_CORS_ORIGINS_ENV, "")
+    origins = parse_cors_origins(cors_origins)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
origins = parse_cors_origins(cors_origins or os.getenv(MCP_CORS_ORIGINS_ENV, ""))
if cors_origins is None:
cors_origins = os.getenv(MCP_CORS_ORIGINS_ENV, "")
origins = parse_cors_origins(cors_origins)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@airbyte/mcp/_http_config.py` at line 45, The CORS origin selection in
build_http_middleware currently treats an explicit empty string the same as no
value, so cors_origins="" falls back to MCP_CORS_ORIGINS unexpectedly. Update
the logic around parse_cors_origins in _http_config so only None triggers the
environment lookup, while an empty string is passed through as an intentional
opt-out; this will keep test_build_http_middleware_without_origins and similar
callers independent of env state.

if not origins:
return []

return [
Middleware(
CORSMiddleware,
allow_origins=origins,
allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_headers=_MCP_CLIENT_HEADERS,
expose_headers=["mcp-protocol-version", "mcp-session-id"],
),
]
8 changes: 8 additions & 0 deletions airbyte/mcp/http_main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@
- `MCP_SERVER_URL`: Public base URL. Used for OIDC redirect callbacks and to
derive the MCP endpoint mount path (serves at `/` when the URL has a path
prefix, otherwise defaults to `/mcp`).
- `MCP_CORS_ORIGINS`: Comma-separated CORS origins for remote MCP clients
(for example, browser-based OAuth flows).
- `OIDC_CONFIG_URL`: Keycloak OIDC discovery URL (enables auth with all three OIDC vars)
- `OIDC_CLIENT_ID`: OIDC client identifier
- `OIDC_CLIENT_SECRET`: OIDC client secret

See `docs/MCP_HTTP_DEPLOYMENT.md` for HTTPS deployment and remote client integration.
"""

from __future__ import annotations
Expand All @@ -21,6 +25,7 @@
import os
from urllib.parse import urlparse

from airbyte.mcp._http_config import build_http_middleware
from airbyte.mcp.server import (
DEFAULT_HTTP_HOST,
DEFAULT_HTTP_PORT,
Expand Down Expand Up @@ -52,12 +57,15 @@ def main() -> None:
mcp_path,
)

middleware = build_http_middleware()

app.run(
transport="streamable-http",
host=DEFAULT_HTTP_HOST,
port=DEFAULT_HTTP_PORT,
path=mcp_path,
stateless_http=True,
middleware=middleware or None,
)


Expand Down
23 changes: 23 additions & 0 deletions deploy/Caddyfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Caddy reverse proxy for the Airbyte Replication MCP server.
#
# Usage (with docker-compose.mcp.yml):
# MCP_PUBLIC_HOST=mcp.example.com docker compose -f deploy/docker-compose.mcp.yml up -d
#
# Caddy obtains and renews TLS certificates automatically for MCP_PUBLIC_HOST.

{$MCP_PUBLIC_HOST} {
# Health checks for load balancers and uptime monitors.
handle /health {
reverse_proxy mcp:8080
}

# Streamable HTTP MCP endpoint (default mount path is /mcp).
handle /mcp* {
reverse_proxy mcp:8080
}

# Fallback for path-stripped deployments (MCP_SERVER_URL with a path prefix).
handle {
reverse_proxy mcp:8080
}
Comment on lines +19 to +22

@coderabbitai coderabbitai Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Strip the prefix before proxying this fallback.

handle { reverse_proxy mcp:8080 } still forwards the original URI, so the documented MCP_SERVER_URL=https://host/airbyte flow will hit mcp:8080/airbyte instead of the root-mounted app. Could we either rewrite the prefix here or drop the path-prefixed deployment claim until that rewrite exists?

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/Caddyfile` around lines 19 - 22, The fallback in Caddyfile still
proxies the original request URI, so the path-prefix deployment flow won’t reach
the root app. Update the fallback `handle` block to strip or rewrite the
configured prefix before `reverse_proxy mcp:8080`, using the same path-prefix
logic expected by `MCP_SERVER_URL`; if that can’t be done here, remove the
path-prefixed deployment claim from the docs/config comments. Refer to the
fallback `handle` and `reverse_proxy` block when making the change.

}
57 changes: 57 additions & 0 deletions deploy/docker-compose.mcp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Self-hosted Airbyte Replication MCP server with automatic HTTPS via Caddy.
#
# Prerequisites:
# - DNS for MCP_PUBLIC_HOST must point to this host
# - A dotenv secrets file at ~/.mcp/airbyte_mcp.env (or override MCP_ENV_FILE)
#
# Usage:
# export MCP_PUBLIC_HOST=mcp.example.com
# docker compose -f deploy/docker-compose.mcp.yml up -d --build
#
# Public MCP endpoint (default mount): https://$MCP_PUBLIC_HOST/mcp

services:
mcp:
build:
context: ..
dockerfile: Dockerfile.mcp
restart: unless-stopped
environment:
AIRBYTE_MCP_ENV_FILE: /secrets/airbyte_mcp.env
MCP_SERVER_URL: https://${MCP_PUBLIC_HOST:-localhost}
MCP_CORS_ORIGINS: ${MCP_CORS_ORIGINS:-}
OIDC_CONFIG_URL: ${OIDC_CONFIG_URL:-}
Comment on lines +21 to +23

@coderabbitai coderabbitai Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Don’t default the public hostname to localhost.

With automatic HTTPS, {$MCP_PUBLIC_HOST} becomes the certificate name and MCP_SERVER_URL becomes the public origin. Defaulting both to localhost leaves the stack in a state that won’t produce a usable remote endpoint unless every user overrides it manually. Could we require an explicit public hostname instead?

Also applies to: 45-46

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/docker-compose.mcp.yml` around lines 21 - 23, The MCP public origin is
currently falling back to localhost via MCP_SERVER_URL and related public-host
settings, which makes the compose stack unusable for remote access unless
overridden manually. Update the docker-compose MCP configuration to require an
explicit MCP_PUBLIC_HOST instead of defaulting it to localhost, and ensure the
MCP_SERVER_URL and any other public-host dependent values are derived from that
required hostname in the compose file.

OIDC_CLIENT_ID: ${OIDC_CLIENT_ID:-}
OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET:-}
AIRBYTE_CLOUD_MCP_SAFE_MODE: ${AIRBYTE_CLOUD_MCP_SAFE_MODE:-1}
AIRBYTE_CLOUD_MCP_READONLY_MODE: ${AIRBYTE_CLOUD_MCP_READONLY_MODE:-0}
volumes:
- ${MCP_ENV_FILE:-${HOME}/.mcp/airbyte_mcp.env}:/secrets/airbyte_mcp.env:ro
expose:
- "8080"
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health')"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s

caddy:
image: caddy:2-alpine
restart: unless-stopped
ports:
- "80:80"
- "443:443"
environment:
MCP_PUBLIC_HOST: ${MCP_PUBLIC_HOST:-localhost}
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
depends_on:
mcp:
condition: service_healthy

volumes:
caddy_data:
caddy_config:
10 changes: 7 additions & 3 deletions docs/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,13 +137,17 @@ poe mcp-tool-test list_deployed_cloud_connections '{}'
You can also invoke the server using one of these helper tasks:

```bash
poe mcp-serve-local # STDIO transport (default)
poe mcp-serve-http # HTTP transport on localhost:8000
poe mcp-serve-sse # Server-Sent Events transport on localhost:8000
poe mcp-serve-local # STDIO transport (default)
poe mcp-serve-http # HTTP transport on localhost:8000
poe mcp-serve-sse # Server-Sent Events transport on localhost:8000
poe mcp-serve-streamable-http # Streamable HTTP on localhost:8080 (production transport)

poe mcp-inspect # Show all available MCP tools and their schemas
```

For hosted HTTPS deployment (remote MCP clients), see
[docs/MCP_HTTP_DEPLOYMENT.md](./MCP_HTTP_DEPLOYMENT.md).

### Generating Markdown docs for the MCP Server

The repo ships a small script (`scripts/generate_mcp_markdown.py`) that
Expand Down
Loading
Loading