Redesign cli-proxy: connect to external DIFC proxy started by compiler (#1809)

* Initial plan

* feat: redesign cli-proxy to connect to external DIFC proxy

Replace internal mcpg container management with external DIFC proxy
connection. The gh-aw compiler now starts mcpg on the host; AWF only
launches the cli-proxy container and connects via TCP tunnel.

New CLI flags:
- --difc-proxy-host <host:port>: connect to external DIFC proxy
- --difc-proxy-ca-cert <path>: TLS CA cert from DIFC proxy

Removed CLI flags:
- --enable-cli-proxy
- --cli-proxy-writable
- --cli-proxy-policy
- --cli-proxy-mcpg-image

Key changes:
- Remove mcpg service from Docker Compose generation
- cli-proxy gets its own IP (172.30.0.50) with extra_hosts
- TCP tunnel forwards localhost to host DIFC proxy for TLS
- Write control delegated to DIFC guard policy
- Remove cli-proxy-tls named volume (CA cert bind-mounted)
- Update all tests (1374 pass)

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/a38ec447-a23c-4104-be85-bd7328c48559

* docs: update gh-cli-proxy-design.md for external DIFC proxy

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/a38ec447-a23c-4104-be85-bd7328c48559

* fix: address review feedback for cli-proxy redesign

- Add port validation and error logging to tcp-tunnel.js
- Make NODE_EXTRA_CA_CERTS conditional on cert existence
- Fix CLI help text for --difc-proxy-ca-cert

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/a38ec447-a23c-4104-be85-bd7328c48559

* fix: apply PR review feedback for cli-proxy redesign

- Use URL-based parser for difcProxyHost (IPv6 support)
- Add host iptables rule for cli-proxy → DIFC proxy port
- Add 'alias' to ALWAYS_DENIED_SUBCOMMANDS (shell exec)
- Fail fast when CA cert not found in entrypoint.sh
- Add server.on('error') handler in tcp-tunnel.js
- Rename --enable-cli-proxy to --difc-proxy in predownload

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/11e4f4ef-6a07-4318-9e6f-aba436bd9590

* refactor: extract parseDifcProxyHost to shared utility

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/11e4f4ef-6a07-4318-9e6f-aba436bd9590

* fix: remove CA cert poll loop, update stale flag comments

The CA cert is now a bind mount (not generated at runtime by mcpg),
so it's immediately available at container start. Replace the 30s
polling loop with a single existence check for faster startup.

Also update stale comments referencing --enable-cli-proxy to
--difc-proxy-host in setup-iptables.sh and gh-cli-proxy-wrapper.sh.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Log injection'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Landon Cox <landon.cox@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Author: 4 people

containers/agent/gh-cli-proxy-wrapper.sh

@@ -2,7 +2,7 @@
 # /usr/local/bin/gh-cli-proxy-wrapper
 # Forwards gh CLI invocations to the CLI proxy sidecar over HTTP.
 # This wrapper is installed at /usr/local/bin/gh in the agent container
-# when --enable-cli-proxy is active, so it takes precedence over any
+# when --difc-proxy-host is active, so it takes precedence over any
 # host-mounted gh binary at /host/usr/bin/gh.
 #
 # Dependencies: curl, jq (both available in the agent container)

containers/agent/setup-iptables.sh

@@ -174,7 +174,7 @@ if [ -n "$AWF_API_PROXY_IP" ]; then
 fi
 
 # Allow traffic to CLI proxy sidecar (when enabled)
-# AWF_CLI_PROXY_IP is set by docker-manager.ts when --enable-cli-proxy is used
+# AWF_CLI_PROXY_IP is set by docker-manager.ts when --difc-proxy-host is used
 if [ -n "$AWF_CLI_PROXY_IP" ]; then
   echo "[iptables] Allow traffic to CLI proxy sidecar (${AWF_CLI_PROXY_IP})..."
   iptables -t nat -A OUTPUT -d "$AWF_CLI_PROXY_IP" -j RETURN

containers/cli-proxy/Dockerfile

@@ -1,10 +1,9 @@
-# CLI Proxy sidecar for AWF - provides gh CLI access via mcpg DIFC proxy
+# CLI Proxy sidecar for AWF - provides gh CLI access via external DIFC proxy
 #
 # This container runs the HTTP exec server (port 11000) that receives gh
-# invocations from the agent container.  The mcpg DIFC proxy runs as a
-# separate docker-compose service (awf-cli-proxy-mcpg) using the official
-# gh-aw-mcpg image directly — no binary extraction needed.  GH_HOST is
-# set to the mcpg container so all gh CLI traffic flows through the proxy.
+# invocations from the agent container.  The DIFC proxy (mcpg) runs
+# externally on the host, started by the gh-aw compiler.  A TCP tunnel
+# forwards localhost traffic to the external proxy for TLS hostname matching.
 FROM node:22-alpine
 
 # Install gh CLI and curl for healthchecks/wrapper
@@ -25,7 +24,7 @@ COPY package*.json ./
 RUN npm ci --omit=dev
 
 # Copy application files
-COPY server.js ./
+COPY server.js tcp-tunnel.js ./
 COPY entrypoint.sh /usr/local/bin/entrypoint.sh
 COPY healthcheck.sh /usr/local/bin/healthcheck.sh
 
@@ -38,7 +37,7 @@ RUN addgroup -S cliproxy && adduser -S cliproxy -G cliproxy
 RUN mkdir -p /var/log/cli-proxy && \
     chown -R cliproxy:cliproxy /var/log/cli-proxy
 
-# Create /tmp/proxy-tls directory owned by cliproxy for shared mcpg TLS certs
+# Create /tmp/proxy-tls directory owned by cliproxy for mounted DIFC proxy CA cert
 RUN mkdir -p /tmp/proxy-tls && chown cliproxy:cliproxy /tmp/proxy-tls
 
 # Switch to non-root user

containers/cli-proxy/entrypoint.sh

@@ -1,56 +1,61 @@
 #!/bin/bash
 # CLI Proxy sidecar entrypoint
 #
-# The mcpg DIFC proxy runs as a separate docker-compose service
-# (awf-cli-proxy-mcpg). This container shares mcpg's network namespace
-# (network_mode: service:cli-proxy-mcpg), so localhost resolves to mcpg.
-# This ensures the TLS cert's SAN (localhost + 127.0.0.1) matches the
-# hostname used by the gh CLI, avoiding TLS hostname verification failures.
+# Connects to an external DIFC proxy (mcpg) started by the gh-aw compiler
+# on the host. Uses a TCP tunnel to forward localhost:${DIFC_PORT} to
+# ${DIFC_HOST}:${DIFC_PORT}, so the gh CLI can connect via localhost
+# (matching the DIFC proxy's TLS cert SAN for localhost/127.0.0.1).
 set -e
 
 echo "[cli-proxy] Starting CLI proxy sidecar..."
 
 NODE_PID=""
+TUNNEL_PID=""
 
-# cli-proxy shares mcpg's network namespace, so mcpg is always at localhost.
-# AWF_MCPG_PORT is set by docker-manager.ts.
-MCPG_PORT="${AWF_MCPG_PORT:-18443}"
+# External DIFC proxy host and port, set by docker-manager.ts
+DIFC_HOST="${AWF_DIFC_PROXY_HOST:-host.docker.internal}"
+DIFC_PORT="${AWF_DIFC_PROXY_PORT:-18443}"
 
-echo "[cli-proxy] mcpg proxy at localhost:${MCPG_PORT}"
+echo "[cli-proxy] External DIFC proxy at ${DIFC_HOST}:${DIFC_PORT}"
 
-# Wait for TLS cert to appear in the shared volume (max 30s)
-echo "[cli-proxy] Waiting for mcpg TLS certificate..."
-i=0
-while [ $i -lt 30 ]; do
-  if [ -f /tmp/proxy-tls/ca.crt ]; then
-    echo "[cli-proxy] TLS certificate available"
-    break
-  fi
-  sleep 1
-  i=$((i + 1))
-done
+# Start the TCP tunnel: localhost:${DIFC_PORT} → ${DIFC_HOST}:${DIFC_PORT}
+# This allows the gh CLI to connect via localhost, matching the cert's SAN.
+echo "[cli-proxy] Starting TCP tunnel: localhost:${DIFC_PORT} → ${DIFC_HOST}:${DIFC_PORT}"
+node /app/tcp-tunnel.js "${DIFC_PORT}" "${DIFC_HOST}" "${DIFC_PORT}" &
+TUNNEL_PID=$!
 
+# Verify CA cert is available (bind-mounted from host by docker-manager.ts).
+# Unlike the old architecture where mcpg generated the cert at runtime, the
+# external DIFC proxy has already created the cert before AWF starts, so the
+# bind mount makes it immediately available — no polling needed.
 if [ ! -f /tmp/proxy-tls/ca.crt ]; then
-  echo "[cli-proxy] ERROR: mcpg TLS certificate not found within 30s"
+  echo "[cli-proxy] ERROR: DIFC proxy TLS certificate not found at /tmp/proxy-tls/ca.crt"
+  echo "[cli-proxy] Ensure --difc-proxy-ca-cert points to a valid CA cert file on the host"
   exit 1
 fi
+echo "[cli-proxy] TLS certificate available"
 
-# Configure gh CLI to route through the mcpg proxy (TLS, self-signed CA)
-# Uses localhost because cli-proxy shares mcpg's network namespace — the
-# self-signed cert's SAN covers localhost, so TLS hostname verification passes.
-export GH_HOST="localhost:${MCPG_PORT}"
-export NODE_EXTRA_CA_CERTS="/tmp/proxy-tls/ca.crt"
+# Configure gh CLI to route through the DIFC proxy via the TCP tunnel
+# Uses localhost because the tunnel makes the DIFC proxy appear on localhost,
+# matching the self-signed cert's SAN.
+export GH_HOST="localhost:${DIFC_PORT}"
 export GH_REPO="${GH_REPO:-$GITHUB_REPOSITORY}"
+# The CA cert is guaranteed to exist at this point (we exit above if missing)
+export NODE_EXTRA_CA_CERTS="/tmp/proxy-tls/ca.crt"
 
-echo "[cli-proxy] gh CLI configured to route through mcpg proxy at ${GH_HOST}"
+echo "[cli-proxy] gh CLI configured to route through DIFC proxy at ${GH_HOST}"
 
-# Cleanup handler: stop the Node HTTP server on signal
+# Cleanup handler: stop the Node HTTP server and TCP tunnel on signal
 cleanup() {
   echo "[cli-proxy] Shutting down..."
   if [ -n "$NODE_PID" ]; then
     kill "$NODE_PID" 2>/dev/null || true
     wait "$NODE_PID" 2>/dev/null || true
   fi
+  if [ -n "$TUNNEL_PID" ]; then
+    kill "$TUNNEL_PID" 2>/dev/null || true
+    wait "$TUNNEL_PID" 2>/dev/null || true
+  fi
 }
 trap 'cleanup; exit 0' INT TERM
 

containers/cli-proxy/server.js

@@ -7,13 +7,14 @@
  *   POST /exec    - Execute a gh CLI command and return stdout/stderr/exitCode
  *
  * Security:
- *   - Subcommand allowlist enforced (read-only mode by default)
  *   - Args are exec'd directly via execFile (no shell, no injection)
  *   - Per-command timeout (default 30s)
  *   - Max output size limit to prevent memory exhaustion
+ *   - Meta-commands (auth, config, extension) are always denied
  *
- * The gh CLI running inside this container has GH_HOST set to the mcpg proxy
- * (localhost:18443), so it never sees GH_TOKEN directly.
+ * The gh CLI running inside this container has GH_HOST set to the DIFC proxy
+ * (localhost:18443 via TCP tunnel), so it never sees GH_TOKEN directly.
+ * Write control is handled by the DIFC guard policy, not by this server.
  */
 
 const http = require('http');
@@ -23,77 +24,26 @@ const CLI_PROXY_PORT = parseInt(process.env.AWF_CLI_PROXY_PORT || '11000', 10);
 const COMMAND_TIMEOUT_MS = parseInt(process.env.AWF_CLI_PROXY_TIMEOUT_MS || '30000', 10);
 const MAX_OUTPUT_BYTES = parseInt(process.env.AWF_CLI_PROXY_MAX_OUTPUT_BYTES || String(10 * 1024 * 1024), 10);
 
-// When AWF_CLI_PROXY_WRITABLE=true, allow write operations
-const WRITABLE_MODE = process.env.AWF_CLI_PROXY_WRITABLE === 'true';
-
-/**
- * Subcommands allowed in read-only mode.
- * These commands only retrieve data and do not modify any GitHub resources.
- *
- * Note: 'api' is intentionally excluded even in read-only mode because it is a raw
- * HTTP passthrough that can perform arbitrary POST/PUT/DELETE mutations via -X/--method.
- * Agents should use typed subcommands (gh issue list, gh pr view, etc.) instead.
- * In writable mode, 'api' is permitted since the operator has explicitly opted in.
- */
-const ALLOWED_SUBCOMMANDS_READONLY = new Set([
-  'browse',
-  'cache',
-  'codespace',
-  'gist',
-  'issue',
-  'label',
-  'org',
-  'pr',
-  'release',
-  'repo',
-  'run',
-  'search',
-  'secret',
-  'variable',
-  'workflow',
-]);
-
-/**
- * Actions that are blocked within their parent subcommand in read-only mode.
- * Maps subcommand -> Set of blocked action verbs.
- */
-const BLOCKED_ACTIONS_READONLY = new Map([
-  // cache: delete is a write operation
-  ['cache', new Set(['delete'])],
-  // codespace: create, delete, edit, stop, ports forward are write operations
-  ['codespace', new Set(['create', 'delete', 'edit', 'stop', 'ports'])],
-  ['gist', new Set(['create', 'delete', 'edit'])],
-  ['issue', new Set(['create', 'close', 'delete', 'edit', 'lock', 'pin', 'reopen', 'transfer', 'unpin'])],
-  ['label', new Set(['create', 'delete', 'edit'])],
-  // org: invite changes org membership
-  ['org', new Set(['invite'])],
-  ['pr', new Set(['checkout', 'close', 'create', 'edit', 'lock', 'merge', 'ready', 'reopen', 'review', 'update-branch'])],
-  ['release', new Set(['create', 'delete', 'delete-asset', 'edit', 'upload'])],
-  ['repo', new Set(['archive', 'create', 'delete', 'edit', 'fork', 'rename', 'set-default', 'sync', 'unarchive'])],
-  ['run', new Set(['cancel', 'delete', 'download', 'rerun'])],
-  ['secret', new Set(['delete', 'set'])],
-  ['variable', new Set(['delete', 'set'])],
-  ['workflow', new Set(['disable', 'enable', 'run'])],
-]);
-
 /**
- * Meta-commands that are always denied, even in write mode.
+ * Meta-commands that are always denied.
  * These modify gh itself rather than GitHub resources.
  */
 const ALWAYS_DENIED_SUBCOMMANDS = new Set([
+  'alias',
   'auth',
   'config',
   'extension',
 ]);
 
 /**
- * Validates the gh CLI arguments against the subcommand allowlist.
+ * Validates the gh CLI arguments.
+ * Write control is handled by the DIFC guard policy — this server only
+ * blocks meta-commands that modify gh CLI itself.
  *
  * @param {string[]} args - The argument array (excluding 'gh' itself)
- * @param {boolean} writable - Whether write operations are permitted
  * @returns {{ valid: boolean, error?: string }}
  */
-function validateArgs(args, writable) {
+function validateArgs(args) {
   if (!Array.isArray(args)) {
     return { valid: false, error: 'args must be an array' };
   }
@@ -105,13 +55,7 @@ function validateArgs(args, writable) {
   }
 
   // Find the subcommand by scanning through args, skipping flags and their values.
-  // Handles patterns like: gh --repo owner/repo pr list
-  // Strategy: when we see --flag (without =), assume the next non-flag-like arg is its value.
-  // We also track the subcommand's index so that subsequent action detection doesn't
-  // accidentally pick up a flag value that happens to equal the subcommand string
-  // (e.g. gh --repo pr pr merge 1 would be wrongly parsed by indexOf).
   let subcommand = null;
-  let subcommandIndex = -1;
   let i = 0;
   while (i < args.length) {
     const arg = args[i];
@@ -125,7 +69,6 @@ function validateArgs(args, writable) {
       }
     } else {
       subcommand = arg;
-      subcommandIndex = i;
       break;
     }
   }
@@ -140,28 +83,6 @@ function validateArgs(args, writable) {
     return { valid: false, error: `Subcommand '${subcommand}' is not permitted` };
   }
 
-  if (!writable) {
-    // Read-only mode: check allowlist
-    if (!ALLOWED_SUBCOMMANDS_READONLY.has(subcommand)) {
-      return { valid: false, error: `Subcommand '${subcommand}' is not allowed in read-only mode. Enable write mode with --cli-proxy-writable.` };
-    }
-
-    // Check action-level blocklist
-    const blockedActions = BLOCKED_ACTIONS_READONLY.get(subcommand);
-    if (blockedActions) {
-      // The action is the first non-flag argument after the subcommand.
-      // Use the tracked subcommandIndex (not indexOf) to avoid false matches when
-      // the subcommand string also appears as a flag value earlier in the args array.
-      const action = args.slice(subcommandIndex + 1).find(a => !a.startsWith('-'));
-      if (action && blockedActions.has(action)) {
-        return {
-          valid: false,
-          error: `Action '${subcommand} ${action}' is not allowed in read-only mode. Enable write mode with --cli-proxy-writable.`,
-        };
-      }
-    }
-  }
-
   return { valid: true };
 }
 
@@ -221,7 +142,7 @@ function sendError(res, statusCode, message) {
  * Handle GET /health
  */
 function handleHealth(res) {
-  const body = JSON.stringify({ status: 'ok', service: 'cli-proxy', writable: WRITABLE_MODE });
+  const body = JSON.stringify({ status: 'ok', service: 'cli-proxy' });
   res.writeHead(200, {
     'Content-Type': 'application/json',
     'Content-Length': Buffer.byteLength(body),
@@ -261,7 +182,7 @@ async function handleExec(req, res) {
   const { args, cwd, stdin, env: extraEnv } = body;
 
   // Validate args
-  const validation = validateArgs(args, WRITABLE_MODE);
+  const validation = validateArgs(args);
   if (!validation.valid) {
     return sendError(res, 403, validation.error);
   }
@@ -364,7 +285,7 @@ if (require.main === module) {
   });
 
   server.listen(CLI_PROXY_PORT, '0.0.0.0', () => {
-    console.log(`[cli-proxy] HTTP server listening on port ${CLI_PROXY_PORT} (writable=${WRITABLE_MODE})`);
+    console.log(`[cli-proxy] HTTP server listening on port ${CLI_PROXY_PORT}`);
   });
 
   server.on('error', err => {
@@ -373,4 +294,4 @@ if (require.main === module) {
   });
 }
 
-module.exports = { validateArgs, ALLOWED_SUBCOMMANDS_READONLY, BLOCKED_ACTIONS_READONLY, ALWAYS_DENIED_SUBCOMMANDS };
+module.exports = { validateArgs, ALWAYS_DENIED_SUBCOMMANDS };