Add per-serverID log files for easier backend troubleshooting (#724)

## Per-ServerID Logging Implementation ✅

This PR implements per-serverID log files to make it easier to
troubleshoot individual MCP servers.

### Completed Tasks

- [x] Explore the repository and understand current logging
infrastructure
- [x] Design per-serverID logging architecture
  - [x] Add new `ServerFileLogger` that manages per-serverID log files
  - [x] Create log files named `{serverID}.log` in the log directory
  - [x] Add helper functions to log with serverID context
- [x] Implement per-serverID file logger
- [x] Create `ServerFileLogger` struct with map of serverID -> file
logger
  - [x] Add `LogWithServer` functions that route to per-serverID files
  - [x] Handle concurrent access with proper locking
- [x] Update existing log calls to use per-serverID logging
  - [x] Update backend-related log calls with serverID context
  - [x] Updated launcher package to use per-serverID logging
  - [x] Updated connection.go stderr logging
  - [x] Updated unified server tool registration logging
  - [x] Keep gateway-wide logs in `mcp-gateway.log`
- [x] Add tests for per-serverID logging
  - [x] Test log file creation per serverID
  - [x] Test concurrent logging to different serverID files
  - [x] Test all log levels
  - [x] Test fallback mode
  - [x] Test multiple initialization
  - [x] Test that unified log is preserved ✨
- [x] Documentation
  - [x] Update logger README with per-serverID logging details
  - [x] Update main README to highlight the feature
  - [x] Update AGENTS.md with logging details
  - [x] Document log directory structure
- [x] Verify changes
  - [x] Build successful
  - [x] All unit tests passing
  - [x] Linter passed ✅ (fixed gofmt issue)
  - [x] Code review completed - addressed feedback
  - [x] Security check (CodeQL) - No vulnerabilities found

### Implementation Summary

**Key Feature - Dual Logging:**
✅ Each per-serverID log function writes to BOTH:
1. Server-specific log file (e.g., `github.log`) - isolated messages
2. Unified `mcp-gateway.log` - all messages with `[serverID]` prefix

This ensures **single-view files are preserved** while adding
per-serverID isolation.

**Log Directory Structure:**
```
/tmp/gh-aw/mcp-logs/
├── mcp-gateway.log    # ALL messages (single-view) ✅
├── github.log         # Only GitHub server logs
├── slack.log          # Only Slack server logs
├── notion.log         # Only Notion server logs
├── gateway.md         # Markdown format
└── rpc-messages.jsonl # RPC messages
```

**API Usage:**
```go
// Writes to both {serverID}.log AND mcp-gateway.log
logger.LogInfoWithServer("github", "backend", "Server started")
logger.LogWarnWithServer("slack", "backend", "Connection timeout")
```

**Testing:**
- 8 comprehensive test cases including test for unified log preservation
- All existing tests continue to pass
- Verified single-view and per-serverID logs work correctly
- CodeQL security check passed with no vulnerabilities
- All linting checks pass ✅

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for
you](https://github.com/github/gh-aw-mcpg/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot)
— coding agent works faster and does higher quality work when set up for
your repo.

Author: lpcox

AGENTS.md

@@ -366,11 +366,23 @@ DEBUG_COLORS=0 DEBUG=* ./awmg --config config.toml
 - `MCP_GATEWAY_PAYLOAD_DIR` - Large payload storage directory (sets default for `--payload-dir` flag, default: `/tmp/jq-payloads`)
 
 **File Logging:**
-- Operational logs are always written to `mcp-gateway.log` in the configured log directory
+- Operational logs are always written to log files in the configured log directory
 - Default log directory: `/tmp/gh-aw/mcp-logs` (configurable via `--log-dir` flag or `MCP_GATEWAY_LOG_DIR` env var)
 - Falls back to stdout if log directory cannot be created
+- **Log Files Created:**
+  - `mcp-gateway.log` - Unified log with all messages
+  - `{serverID}.log` - Per-server logs (e.g., `github.log`, `slack.log`) for easier troubleshooting
+  - `gateway.md` - Markdown-formatted logs for GitHub workflow previews
+  - `rpc-messages.jsonl` - Machine-readable JSONL format for RPC message analysis
 - Logs include: startup, client interactions, backend operations, auth events, errors
 
+**Per-ServerID Logging:**
+- Each backend MCP server gets its own log file for easier troubleshooting
+- Use `LogInfoWithServer`, `LogWarnWithServer`, `LogErrorWithServer`, `LogDebugWithServer` functions
+- Example: `logger.LogInfoWithServer("github", "backend", "Server started successfully")`
+- Logs are written to both the server-specific file and the unified `mcp-gateway.log`
+- Thread-safe concurrent logging with automatic fallback
+
 **Large Payload Handling:**
 - Large tool response payloads are stored in the configured payload directory
 - Default payload directory: `/tmp/jq-payloads` (configurable via `--payload-dir` flag, `MCP_GATEWAY_PAYLOAD_DIR` env var, or `payload_dir` in config)

README.md

@@ -23,6 +23,7 @@ This gateway is used with [GitHub Agentic Workflows](https://github.com/github/g
 - **Stdio Transport**: JSON-RPC 2.0 over stdin/stdout for MCP communication
 - **Container Detection**: Automatic detection of containerized environments with security warnings
 - **Enhanced Debugging**: Detailed error context and troubleshooting suggestions for command failures
+- **Per-ServerID Logs**: Separate log files for each backend MCP server (`{serverID}.log`) for easier troubleshooting
 
 ## Getting Started
 
@@ -67,6 +68,10 @@ For detailed setup instructions, building from source, and local development, se
 - `-e MCP_GATEWAY_*`: Required environment variables
 - `-v /var/run/docker.sock`: Required for spawning backend MCP servers
 - `-v /path/to/logs:/tmp/gh-aw/mcp-logs`: Mount for persistent gateway logs (or use `-e MCP_GATEWAY_LOG_DIR=/custom/path` with matching volume mount)
+  - `mcp-gateway.log`: Unified log with all messages
+  - `{serverID}.log`: Per-server logs for easier troubleshooting
+  - `gateway.md`: Markdown-formatted logs for GitHub workflow previews
+  - `rpc-messages.jsonl`: Machine-readable RPC message logs
 - `-p 8000:8000`: Port mapping must match `MCP_GATEWAY_PORT`
 
 MCPG will start in routed mode on `http://0.0.0.0:8000` (using `MCP_GATEWAY_PORT`), proxying MCP requests to your configured backend servers.
@@ -329,15 +334,44 @@ MCP_GATEWAY_PORT=3000 ./run.sh
 
 MCPG provides comprehensive logging of all gateway operations to help diagnose issues and monitor activity.
 
+### Log Files
+
+The gateway creates multiple log files for different purposes:
+
+1. **`mcp-gateway.log`** - Unified log with all gateway messages
+2. **`{serverID}.log`** - Per-server logs (e.g., `github.log`, `slack.log`) for easier troubleshooting of specific backend servers
+3. **`gateway.md`** - Markdown-formatted logs for GitHub workflow previews
+4. **`rpc-messages.jsonl`** - Machine-readable JSONL format for RPC message analysis
+
 ### Log File Location
 
-By default, logs are written to `/tmp/gh-aw/mcp-logs/mcp-gateway.log`. This location can be configured using either:
+By default, logs are written to `/tmp/gh-aw/mcp-logs/`. This location can be configured using either:
 
 1. **`MCP_GATEWAY_LOG_DIR` environment variable** - Sets the default log directory
 2. **`--log-dir` flag** - Overrides the environment variable and default
 
 The precedence order is: `--log-dir` flag → `MCP_GATEWAY_LOG_DIR` env var → default (`/tmp/gh-aw/mcp-logs`)
 
+### Per-ServerID Logs
+
+Each backend MCP server gets its own log file (e.g., `github.log`, `slack.log`) in addition to the unified `mcp-gateway.log`. This makes it much easier to:
+
+- Debug issues with a specific backend server
+- View all activity for one server without filtering
+- Identify which server is causing problems
+- Troubleshoot server-specific configuration issues
+
+Example log directory structure:
+```
+/tmp/gh-aw/mcp-logs/
+├── mcp-gateway.log    # All messages
+├── github.log         # Only GitHub server logs
+├── slack.log          # Only Slack server logs
+├── notion.log         # Only Notion server logs
+├── gateway.md         # Markdown format
+└── rpc-messages.jsonl # RPC messages
+```
+
 **Using the environment variable:**
 ```bash
 export MCP_GATEWAY_LOG_DIR=/var/log/mcp-gateway

internal/cmd/root.go

@@ -138,6 +138,12 @@ func run(cmd *cobra.Command, args []string) error {
 	}
 	defer logger.CloseGlobalLogger()
 
+	// Initialize per-serverID logger
+	if err := logger.InitServerFileLogger(logDir); err != nil {
+		log.Printf("Warning: Failed to initialize server file logger: %v", err)
+	}
+	defer logger.CloseServerFileLogger()
+
 	// Initialize markdown logger for GitHub workflow preview
 	if err := logger.InitMarkdownLogger(logDir, "gateway.md"); err != nil {
 		log.Printf("Warning: Failed to initialize markdown logger: %v", err)

internal/launcher/launcher.go

@@ -63,14 +63,14 @@ func New(ctx context.Context, cfg *config.Config) *Launcher {
 
 // GetOrLaunch returns an existing connection or launches a new one
 func GetOrLaunch(l *Launcher, serverID string) (*mcp.Connection, error) {
-	logger.LogDebug("backend", "GetOrLaunch called for server: %s", serverID)
+	logger.LogDebugWithServer(serverID, "backend", "GetOrLaunch called for server: %s", serverID)
 	logLauncher.Printf("GetOrLaunch called: serverID=%s", serverID)
 
 	// Check if already exists
 	l.mu.RLock()
 	if conn, ok := l.connections[serverID]; ok {
 		l.mu.RUnlock()
-		logger.LogDebug("backend", "Reusing existing backend connection: %s", serverID)
+		logger.LogDebugWithServer(serverID, "backend", "Reusing existing backend connection: %s", serverID)
 		logLauncher.Printf("Reusing existing connection: serverID=%s", serverID)
 		return conn, nil
 	}
@@ -84,37 +84,37 @@ func GetOrLaunch(l *Launcher, serverID string) (*mcp.Connection, error) {
 
 	// Double-check after acquiring write lock
 	if conn, ok := l.connections[serverID]; ok {
-		logger.LogDebug("backend", "Backend connection created by another goroutine: %s", serverID)
+		logger.LogDebugWithServer(serverID, "backend", "Backend connection created by another goroutine: %s", serverID)
 		logLauncher.Printf("Connection created by another goroutine: serverID=%s", serverID)
 		return conn, nil
 	}
 
 	// Get server config
 	serverCfg, ok := l.config.Servers[serverID]
 	if !ok {
-		logger.LogError("backend", "Backend server not found in config: %s", serverID)
+		logger.LogErrorWithServer(serverID, "backend", "Backend server not found in config: %s", serverID)
 		logLauncher.Printf("Server not found in config: serverID=%s", serverID)
 		return nil, fmt.Errorf("server '%s' not found in config", serverID)
 	}
 	logLauncher.Printf("Retrieved server config: serverID=%s, type=%s", serverID, serverCfg.Type)
 
 	// Handle HTTP backends differently
 	if serverCfg.Type == "http" {
-		logger.LogInfo("backend", "Configuring HTTP MCP backend: %s, url=%s", serverID, serverCfg.URL)
+		logger.LogInfoWithServer(serverID, "backend", "Configuring HTTP MCP backend: %s, url=%s", serverID, serverCfg.URL)
 		log.Printf("[LAUNCHER] Configuring HTTP MCP backend: %s", serverID)
 		log.Printf("[LAUNCHER] URL: %s", serverCfg.URL)
 		logLauncher.Printf("HTTP backend: serverID=%s, url=%s", serverID, serverCfg.URL)
 
 		// Create an HTTP connection
 		conn, err := mcp.NewHTTPConnection(l.ctx, serverID, serverCfg.URL, serverCfg.Headers)
 		if err != nil {
-			logger.LogError("backend", "Failed to create HTTP connection: %s, error=%v", serverID, err)
+			logger.LogErrorWithServer(serverID, "backend", "Failed to create HTTP connection: %s, error=%v", serverID, err)
 			log.Printf("[LAUNCHER] ❌ FAILED to create HTTP connection for '%s'", serverID)
 			log.Printf("[LAUNCHER] Error: %v", err)
 			return nil, fmt.Errorf("failed to create HTTP connection: %w", err)
 		}
 
-		logger.LogInfo("backend", "Successfully configured HTTP MCP backend: %s", serverID)
+		logger.LogInfoWithServer(serverID, "backend", "Successfully configured HTTP MCP backend: %s", serverID)
 		log.Printf("[LAUNCHER] Successfully configured HTTP backend: %s", serverID)
 		logLauncher.Printf("HTTP connection configured: serverID=%s", serverID)
 
@@ -178,7 +178,7 @@ func GetOrLaunch(l *Launcher, serverID string) (*mcp.Connection, error) {
 // GetOrLaunchForSession returns a session-aware connection or launches a new one
 // This is used for stateful stdio backends that require persistent connections
 func GetOrLaunchForSession(l *Launcher, serverID, sessionID string) (*mcp.Connection, error) {
-	logger.LogDebug("backend", "GetOrLaunchForSession called: server=%s, session=%s", serverID, sessionID)
+	logger.LogDebugWithServer(serverID, "backend", "GetOrLaunchForSession called: server=%s, session=%s", serverID, sessionID)
 	logLauncher.Printf("GetOrLaunchForSession called: serverID=%s, sessionID=%s", serverID, sessionID)
 
 	// Get server config first to determine backend type
@@ -187,7 +187,7 @@ func GetOrLaunchForSession(l *Launcher, serverID, sessionID string) (*mcp.Connec
 	l.mu.RUnlock()
 
 	if !ok {
-		logger.LogError("backend", "Backend server not found in config: %s", serverID)
+		logger.LogErrorWithServer(serverID, "backend", "Backend server not found in config: %s", serverID)
 		return nil, fmt.Errorf("server '%s' not found in config", serverID)
 	}
 
@@ -200,7 +200,7 @@ func GetOrLaunchForSession(l *Launcher, serverID, sessionID string) (*mcp.Connec
 	logLauncher.Printf("Checking session pool: serverID=%s, sessionID=%s", serverID, sessionID)
 	// For stdio backends, check the session pool first
 	if conn, exists := l.sessionPool.Get(serverID, sessionID); exists {
-		logger.LogDebug("backend", "Reusing session connection: server=%s, session=%s", serverID, sessionID)
+		logger.LogDebugWithServer(serverID, "backend", "Reusing session connection: server=%s, session=%s", serverID, sessionID)
 		logLauncher.Printf("Reusing session connection: serverID=%s, sessionID=%s", serverID, sessionID)
 		return conn, nil
 	}
@@ -214,7 +214,7 @@ func GetOrLaunchForSession(l *Launcher, serverID, sessionID string) (*mcp.Connec
 
 	// Double-check after acquiring lock
 	if conn, exists := l.sessionPool.Get(serverID, sessionID); exists {
-		logger.LogDebug("backend", "Session connection created by another goroutine: server=%s, session=%s", serverID, sessionID)
+		logger.LogDebugWithServer(serverID, "backend", "Session connection created by another goroutine: server=%s, session=%s", serverID, sessionID)
 		logLauncher.Printf("Session connection created by another goroutine: serverID=%s, sessionID=%s", serverID, sessionID)
 		return conn, nil
 	}

internal/launcher/log_helpers.go

@@ -21,7 +21,7 @@ func sessionSuffix(sessionID string) string {
 
 // logSecurityWarning logs container security warnings
 func (l *Launcher) logSecurityWarning(serverID string, serverCfg *config.ServerConfig) {
-	logger.LogWarn("backend", "Server '%s' uses direct command execution inside a container (command: %s)", serverID, serverCfg.Command)
+	logger.LogWarnWithServer(serverID, "backend", "Server '%s' uses direct command execution inside a container (command: %s)", serverID, serverCfg.Command)
 	log.Printf("[LAUNCHER] ⚠️  WARNING: Server '%s' uses direct command execution inside a container", serverID)
 	log.Printf("[LAUNCHER] ⚠️  Security Notice: Command '%s' will execute with the same privileges as the gateway", serverCfg.Command)
 	log.Printf("[LAUNCHER] ⚠️  Consider using 'container' field instead for better isolation")
@@ -30,11 +30,11 @@ func (l *Launcher) logSecurityWarning(serverID string, serverCfg *config.ServerC
 // logLaunchStart logs server launch initiation
 func (l *Launcher) logLaunchStart(serverID, sessionID string, serverCfg *config.ServerConfig, isDirectCommand bool) {
 	if sessionID != "" {
-		logger.LogInfo("backend", "Launching MCP backend server for session: server=%s, session=%s, command=%s, args=%v", serverID, sessionID, serverCfg.Command, sanitize.SanitizeArgs(serverCfg.Args))
+		logger.LogInfoWithServer(serverID, "backend", "Launching MCP backend server for session: server=%s, session=%s, command=%s, args=%v", serverID, sessionID, serverCfg.Command, sanitize.SanitizeArgs(serverCfg.Args))
 		log.Printf("[LAUNCHER] Starting MCP server for session: %s (session: %s)", serverID, sessionID)
 		logLauncher.Printf("Launching new session server: serverID=%s, sessionID=%s, command=%s", serverID, sessionID, serverCfg.Command)
 	} else {
-		logger.LogInfo("backend", "Launching MCP backend server: %s, command=%s, args=%v", serverID, serverCfg.Command, sanitize.SanitizeArgs(serverCfg.Args))
+		logger.LogInfoWithServer(serverID, "backend", "Launching MCP backend server: %s, command=%s, args=%v", serverID, serverCfg.Command, sanitize.SanitizeArgs(serverCfg.Args))
 		log.Printf("[LAUNCHER] Starting MCP server: %s", serverID)
 		logLauncher.Printf("Launching new server: serverID=%s, command=%s, inContainer=%v, isDirectCommand=%v",
 			serverID, serverCfg.Command, l.runningInContainer, isDirectCommand)
@@ -70,7 +70,7 @@ func (l *Launcher) logEnvPassthrough(args []string) {
 
 // logLaunchError logs detailed launch failure diagnostics
 func (l *Launcher) logLaunchError(serverID, sessionID string, err error, serverCfg *config.ServerConfig, isDirectCommand bool) {
-	logger.LogError("backend", "Failed to launch MCP backend server%s: server=%s%s, error=%v",
+	logger.LogErrorWithServer(serverID, "backend", "Failed to launch MCP backend server%s: server=%s%s, error=%v",
 		sessionSuffix(sessionID), serverID, sessionSuffix(sessionID), err)
 	log.Printf("[LAUNCHER] ❌ FAILED to launch server '%s'%s", serverID, sessionSuffix(sessionID))
 	log.Printf("[LAUNCHER] Error: %v", err)
@@ -97,7 +97,7 @@ func (l *Launcher) logLaunchError(serverID, sessionID string, err error, serverC
 
 // logTimeoutError logs startup timeout diagnostics
 func (l *Launcher) logTimeoutError(serverID, sessionID string) {
-	logger.LogError("backend", "MCP backend server startup timeout%s: server=%s%s, timeout=%v",
+	logger.LogErrorWithServer(serverID, "backend", "MCP backend server startup timeout%s: server=%s%s, timeout=%v",
 		sessionSuffix(sessionID), serverID, sessionSuffix(sessionID), l.startupTimeout)
 	log.Printf("[LAUNCHER] ❌ Server startup timed out after %v", l.startupTimeout)
 	log.Printf("[LAUNCHER] ⚠️  The server may be hanging or taking too long to initialize")
@@ -112,11 +112,11 @@ func (l *Launcher) logTimeoutError(serverID, sessionID string) {
 // logLaunchSuccess logs successful server launch
 func (l *Launcher) logLaunchSuccess(serverID, sessionID string) {
 	if sessionID != "" {
-		logger.LogInfo("backend", "Successfully launched MCP backend server for session: server=%s, session=%s", serverID, sessionID)
+		logger.LogInfoWithServer(serverID, "backend", "Successfully launched MCP backend server for session: server=%s, session=%s", serverID, sessionID)
 		log.Printf("[LAUNCHER] Successfully launched: %s (session: %s)", serverID, sessionID)
 		logLauncher.Printf("Session connection established: serverID=%s, sessionID=%s", serverID, sessionID)
 	} else {
-		logger.LogInfo("backend", "Successfully launched MCP backend server: %s", serverID)
+		logger.LogInfoWithServer(serverID, "backend", "Successfully launched MCP backend server: %s", serverID)
 		log.Printf("[LAUNCHER] Successfully launched: %s", serverID)
 		logLauncher.Printf("Connection established: serverID=%s", serverID)
 	}

internal/logger/README.md

@@ -11,6 +11,61 @@ A simple, debug-style logging framework for Go that follows the pattern matching
 - **Automatic color coding**: Each namespace gets a unique color when stderr is a TTY
 - **Zero overhead**: Logger enabled state is computed once at construction time
 - **Thread-safe**: Safe for concurrent use
+- **Per-ServerID Logs**: Separate log files for each backend MCP server for easier troubleshooting
+
+## Per-ServerID Logging
+
+The logger package supports creating separate log files for each backend MCP server (identified by serverID). This makes it much easier to troubleshoot specific servers without sifting through unified logs.
+
+### How It Works
+
+- Each serverID gets its own log file: `{serverID}.log` in the log directory
+- Logs are also written to the main `mcp-gateway.log` for a unified view
+- Concurrent writes to different serverID logs are handled safely
+- Fallback to unified logging if per-serverID logging cannot be initialized
+
+### Usage
+
+```go
+import "github.com/github/gh-aw-mcpg/internal/logger"
+
+// Log to both the unified log and the server-specific log
+logger.LogInfoWithServer("github", "backend", "Server started successfully")
+logger.LogWarnWithServer("slack", "backend", "Connection timeout")
+logger.LogErrorWithServer("github", "backend", "Failed to authenticate: %v", err)
+logger.LogDebugWithServer("notion", "backend", "Processing request: %v", req)
+```
+
+### Log File Structure
+
+When per-serverID logging is enabled, the log directory contains:
+```
+/tmp/gh-aw/mcp-logs/
+├── mcp-gateway.log    # Unified log with all messages
+├── github.log          # Only logs for the "github" server
+├── slack.log           # Only logs for the "slack" server
+└── notion.log          # Only logs for the "notion" server
+```
+
+Each server-specific log file contains only the messages related to that serverID, making it easier to debug issues with individual backend servers.
+
+### Initialization
+
+Per-serverID logging is automatically initialized when the gateway starts:
+
+```go
+// In internal/cmd/root.go
+logger.InitServerFileLogger(logDir)
+defer logger.CloseServerFileLogger()
+```
+
+### Benefits
+
+- **Easier Debugging**: View all logs for a specific server in isolation
+- **Reduced Noise**: No need to filter through logs from other servers
+- **Better Troubleshooting**: Quickly identify which server is having issues
+- **Concurrent Access**: Safe to log to multiple servers simultaneously
+- **Backward Compatible**: Falls back gracefully if initialization fails
 
 ## Usage