Commit 11b01ec
* feat: generate mcp-config.json from MCPG runtime output instead of compile time
Replaces the compile-time generate_mcp_client_config() approach with runtime
conversion of MCPG's actual gateway output, matching gh-aw's
convert_gateway_config_copilot.cjs pattern.
Changes:
- Remove generate_mcp_client_config() and {{ mcp_client_config }} template marker
- MCPG stdout is now captured to gateway-output.json
- After health check, poll until gateway output contains valid JSON
- Use jq to transform URLs (127.0.0.1 -> host.docker.internal) and add tools: [*]
- Apply same changes to both standalone and 1ES templates
- Update AGENTS.md documentation
This ensures the Copilot CLI config reflects MCPG's actual runtime state rather
than a compile-time prediction, fixing agents not seeing configured MCPs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: create /tmp/gh-aw/mcp-logs before redirecting MCPG stderr
The stderr redirect to /tmp/gh-aw/mcp-logs/stderr.log failed because the
directory didn't exist yet. Add it to the mkdir -p call alongside the
gateway output directory.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: stream MCPG stderr to pipeline console via tee
Use process substitution to tee stderr to both the log file and the
pipeline console, giving real-time visibility into MCPG startup logs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: copy MCPG logs to published artifact directory
Copy /tmp/gh-aw/mcp-logs/ into staging/logs/mcpg/ so MCPG gateway logs,
stderr output, and gateway-output.json are available in the published
pipeline artifact for post-run debugging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: pre-install Azure DevOps MCP to avoid MCPG startup timeout
MCPG has a 30s startup timeout for stdio MCP containers. The previous
approach used npx -y @azure-devops/mcp which downloads the package at
container launch time, regularly exceeding this timeout.
The AzureDevOpsExtension now emits a prepare_steps() that builds a local
Docker image (ado-mcp-cached:latest) with @azure-devops/mcp pre-installed
via npm install -g. The MCPG config references this cached image, so the
container starts instantly without any network download.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: add node ecosystem domains for Azure DevOps MCP container
The ADO MCP container runs via npx which needs npm registry access to
resolve @azure-devops/mcp. AWF's host-level iptables rules block outbound
traffic from MCPG-spawned containers unless the domains are allowlisted.
Add 'node' ecosystem identifier to AzureDevOpsExtension::required_hosts()
so npm/node domains are included in AWF's allowed domains list.
Also reverts the cached image approach (pre-install via docker commit)
as allowing network access is the correct fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: add MCPG debug logging and backend probe step
Two changes to surface MCPG failures early:
1. Pass DEBUG=* to the MCPG container to enable full debug logging
on stderr (streamed live via tee). This surfaces backend launch
details, connection errors, and timeout diagnostics.
2. Add a 'Verify MCP backends' step after MCPG starts that probes
each configured backend with a tools/list call. This forces
MCPG's lazy initialization and catches failures (e.g., container
timeout, network blocked) before the agent runs. Failures are
surfaced as ADO pipeline warnings.
Relates to #254
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: capture HTTP status and body in MCP backend probe
The probe was using curl -sf which swallows error responses (exit 22
on 4xx). Now captures HTTP status code and response body separately
so we can see what MCPG actually returns on failure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: pass MCPG API key via env mapping in probe step
Secret variables set via ##vso[task.setvariable;issecret=true] must be
explicitly mapped via env: to be available in bash steps. The probe was
using \ macro syntax in the script body, but bash
interprets \ as command substitution before ADO can expand it.
Pass the key via env: mapping as MCPG_API_KEY and reference it as
\ in the script.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: use raw API key for MCPG auth (not Bearer scheme)
MCPG spec 7.1 requires the API key directly in the Authorization
header, not as a Bearer token. The probe was sending
'Authorization: Bearer <key>' but MCPG expects 'Authorization: <key>'.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: perform MCP initialize handshake before tools/list probe
MCP requires an initialize handshake before any other method. The probe
now sends initialize first, captures the Mcp-Session-Id from the response
header, then sends tools/list with that session ID. Also parses SSE
data lines to extract the tool count.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: add --network host to ADO MCP container for AWF compatibility
AWF's DOCKER-USER iptables rules block outbound traffic from containers
on Docker's default bridge network. The ADO MCP container (spawned by
MCPG as a sibling container) was unable to reach dev.azure.com, causing
MCP error -32001 (request timed out) on every tool call.
Adding --network host via the args field bypasses the FORWARD chain
rules, matching gh-aw's approach for its built-in agentic-workflows
MCP server (see mcp_config_builtin.go).
SafeOutputs was unaffected because it runs as an HTTP backend on
localhost — no FORWARD chain involved.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: guard MCPG debug diagnostics behind --debug-pipeline flag
Move debug/diagnostic pipeline additions behind a compile-time flag
(--debug-pipeline, debug builds only) matching the --skip-integrity
pattern:
- MCPG DEBUG=* env var and stderr tee → {{ mcpg_debug_flags }} marker
- MCP backend probe step → {{ verify_mcp_backends }} marker
When --debug-pipeline is not passed (the default), both markers resolve
to empty strings and the generated pipeline contains no debug
diagnostics. MCPG log artifact collection is kept always-on as it is
low-cost and useful in production.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: add DEBUG=* env to ADO MCP container for diagnostics
The ADO MCP container still times out on API calls despite --network host.
Adding DEBUG=* enables verbose logging from the @azure-devops/mcp npm
package to surface the root cause (auth issues, DNS, proxy, etc.).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test: add integration tests for --skip-integrity and --debug-pipeline
Add 12 new compiler integration tests covering:
- --skip-integrity: omits integrity step, produces valid YAML (both targets)
- --debug-pipeline: includes DEBUG env, stderr tee, probe step, produces
valid YAML (both targets)
- Combined flags work together
- Default (no flags) excludes debug content and includes integrity step
- Probe step indentation is correct for both standalone (8sp) and 1ES (18sp)
Also fix replacement content indentation: replace_with_indent adds the
marker's indent to continuation lines, so replacement content must have
zero base indent.
Also refactor compile_fixture into compile_fixture_with_flags with
atomic counter for unique temp dirs to prevent parallel test collisions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: resolve debug step replacement ordering and add marker test
Debug replacements must run BEFORE extra_replacements so that
{{ mcpg_port }} inside the probe step content gets resolved by the
subsequent extra_replacements pass. Previously, debug replacements
ran after extra_replacements, leaving {{ mcpg_port }} unresolved.
Add test_debug_pipeline_no_unresolved_markers to catch this class of
ordering bugs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: fix docker run line continuation and simplify debug markers
Fix bash line continuation in the MCPG docker run command:
- mcpg_debug_flags emits backslash when disabled (maintains line
continuation) and -e DEBUG env flag when enabled
- Stderr tee is always-on (baked into template), not debug-gated
- Fix replacement ordering: debug before extra_replacements so
mcpg_port in probe step content gets resolved
- Drop mcpg_stderr_redirect marker (back to 2 markers total)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: generic pipeline variable injection for MCP extensions
Replace the ADO-specific special case in generate_mcpg_docker_env with
a generic mechanism driven by the CompilerExtension trait.
New trait method: required_pipeline_vars() -> Vec<PipelineEnvMapping>
- Extensions declare which container env vars map to pipeline variables
- AzureDevOpsExtension maps AZURE_DEVOPS_EXT_PAT -> SC_READ_TOKEN
The compiler generates the full chain generically:
1. env: block on MCPG step (ADO secret -> bash var)
2. -e flag on MCPG docker run (bash var -> MCPG process env)
3. MCPG config keeps empty string (MCPG passthrough to child)
ADO secrets require env: mapping because Azure DevOps does not expand
secret variables via macro syntax in script bodies.
Also updates azure-devops-mcp-agent fixture to use first-class
tools.azure-devops instead of manual mcp-servers configuration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: use correct auth for @azure-devops/mcp (envvar + ADO_MCP_AUTH_TOKEN)
The @azure-devops/mcp npm package does NOT use AZURE_DEVOPS_EXT_PAT
(that's for the az devops CLI extension). It accepts auth type via
CLI arg (-a) and reads the token from a specific env var per type.
For pipeline use with bearer tokens from az account get-access-token:
- Add '-a envvar' to entrypoint args (selects envvar auth mode)
- Map ADO_MCP_AUTH_TOKEN (not AZURE_DEVOPS_EXT_PAT) to SC_READ_TOKEN
- The ADO MCP reads the bearer token from ADO_MCP_AUTH_TOKEN
Auth types in @azure-devops/mcp (src/auth.ts):
- 'pat': reads PERSONAL_ACCESS_TOKEN (base64 PAT)
- 'envvar': reads ADO_MCP_AUTH_TOKEN (bearer token) ← we use this
- 'azcli'/'env': uses DefaultAzureCredential
- 'interactive': OAuth browser flow (default, breaks stdio)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat(run): fix MCPG local development issues
- Use dynamic high port for MCPG instead of hardcoded port 80 (requires
admin privileges on most systems)
- Platform-aware Docker networking: --network host on Linux, -p port
mapping on Windows/macOS where Docker Desktop uses a VM
- Rewrite SafeOutputs URL to host.docker.internal on Windows/macOS so
MCPG container can reach host services
- Derive MCP client config from MCPG runtime gateway output instead of
compile-time prediction (matches pipeline behavior)
- Add AdoAuthMode enum to AzureDevOpsExtension: Bearer for pipelines
(ADO_MCP_AUTH_TOKEN), Pat for local dev (PERSONAL_ACCESS_TOKEN)
- Remove stale generate_mcp_client_config re-export (removed on branch)
- Update --pat help text to describe MCPG token plumbing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat(run): add MCPG and Copilot CLI debug capabilities to local run mode
- Capture MCPG stderr: inherit to terminal in debug mode (-d),
redirect to stderr.log in normal mode (previously discarded via
Stdio::null)
- Pass DEBUG=* env to MCPG container when -d flag is set, enabling
full namespace-based debug logging ([LAUNCHER] output, [stderr]
from spawned containers, protocol diagnostics)
- Remove DEBUG=* from ADO MCP server config env — MCPG forwards
server env to child containers where it corrupts JSON-RPC stdio
- Surface MCPG log file locations after startup with tail -f tip
- Dump MCPG diagnostic logs (stderr.log, mcp-gateway.log, per-server
logs) on startup failure for actionable error messages
- Probe MCP backends in debug mode: send initialize + tools/list
handshakes to each backend after gateway ready, catching broken
MCPs before the agent runs
- Pass --log-level debug and --log-dir to Copilot CLI when -d flag
is set for verbose agent logging
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(run): base64-encode PAT for ADO MCP and pass prompts via @file
- Encode raw PAT as base64(:<PAT>) for PERSONAL_ACCESS_TOKEN env var,
matching the microsoft/azure-devops-mcp -a pat auth contract
- Replace hardcoded debug prompt with @filepath reference to the
agent-prompt.md file already written to disk
- Update standalone and 1ES pipeline templates to use
--prompt @/tmp/awf-tools/agent-prompt.md instead of
--prompt \\\\, avoiding shell expansion issues
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: address PR review feedback
- Add missing #[test] attribute on test_debug_pipeline_probe_step_indentation_standalone
- Fix pat doc comment: PERSONAL_ACCESS_TOKEN, not ADO_MCP_AUTH_TOKEN
- Prevent dangling env: null on MCPG step when no extensions need
pipeline vars — generate_mcpg_step_env now emits the full env: block
or empty string
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: remove stray list_projects.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* refactor: cleanup duplicate comment, mcpg log dir, and extension API
- Remove duplicate step 12 comment in compile_pipeline (common.rs)
- Use output_dir for mcpg-logs in both debug and non-debug modes,
preventing stray directories in the user's working directory
- Collapse collect_extensions_with_ado_auth one-liner wrapper into
collect_extensions_with_auth, eliminating the intermediate
collect_extensions_with_options private function
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent d20ae25 commit 11b01ec
12 files changed
Lines changed: 1290 additions & 410 deletions
File tree
- src
- compile
- data
- tests
- fixtures
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
739 | 739 | | |
740 | 740 | | |
741 | 741 | | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
742 | 754 | | |
743 | 755 | | |
744 | 756 | | |
| |||
795 | 807 | | |
796 | 808 | | |
797 | 809 | | |
798 | | - | |
| 810 | + | |
799 | 811 | | |
800 | | - | |
801 | | - | |
802 | | - | |
803 | | - | |
804 | | - | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
805 | 820 | | |
806 | | - | |
807 | | - | |
808 | | - | |
809 | | - | |
810 | | - | |
811 | | - | |
812 | | - | |
813 | | - | |
814 | | - | |
815 | | - | |
816 | | - | |
817 | | - | |
818 | | - | |
819 | | - | |
820 | | - | |
821 | | - | |
822 | | - | |
823 | | - | |
824 | | - | |
825 | | - | |
826 | | - | |
827 | | - | |
828 | | - | |
829 | | - | |
830 | | - | |
831 | | - | |
832 | | - | |
| 821 | + | |
833 | 822 | | |
834 | 823 | | |
835 | 824 | | |
| |||
986 | 975 | | |
987 | 976 | | |
988 | 977 | | |
| 978 | + | |
989 | 979 | | |
990 | 980 | | |
991 | 981 | | |
| |||
0 commit comments