Fix documentation discrepancies: payload threshold and internal directories (#889)

Nightly documentation reconciliation identified critical discrepancies
between documented and actual default values, plus incomplete internal
directory listings.

## Changes

### Critical: Payload Size Threshold Default
Updated README.md (3 locations) to reflect actual implementation default
of **10240 bytes (10KB)** instead of incorrectly documented 1024 bytes:
- CLI flag description (line 206)
- Flags usage table (line 281)  
- Environment variables table (line 319)

Implementation sources:
- `internal/cmd/flags_logging.go:14` - `defaultPayloadSizeThreshold =
10240`
- `internal/config/config_payload.go:11` - `DefaultPayloadSizeThreshold
= 10240`

### Minor: Complete Internal Directory Listings
Added missing internal packages to project structure documentation:

**CONTRIBUTING.md**: Added 9 missing directories (auth, difc, envutil,
middleware, sys, testutil, version, tty, timeutil) to both tree diagram
and Key Directories section

**AGENTS.md**: Added 7 missing directories (difc, envutil, middleware,
sys, testutil, tty, version) to project structure list

All 16 internal directories now documented in alphabetical order:
`auth`, `cmd`, `config`, `difc`, `envutil`, `guard`, `launcher`,
`logger`, `mcp`, `middleware`, `server`, `sys`, `testutil`, `timeutil`,
`tty`, `version`

> [!WARNING]
>
> <details>
> <summary>Firewall rules blocked me from connecting to one or more
addresses (expand for details)</summary>
>
> #### I tried to connect to the following addresses, but was blocked by
firewall rules:
>
> - `example.com`
> - Triggering command: `/tmp/go-build2370360416/b279/launcher.test
/tmp/go-build2370360416/b279/launcher.test
-test.testlogfile=/tmp/go-build2370360416/b279/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/runtime/cgo
git ache/go/1.25.6/x64/pkg/tool/linu-o --global r it 08.o -d go git
x_amd64/vet
e/REDACTED/work/gh/opt/hostedtoolcache/go/1.25.6/x64/pkg/tool/linux_amd64/vet
64/src/runtime/c-unsafeptr=false 64/bin/git x_amd64/vet` (dns block)
> - `invalid-host-that-does-not-exist-12345.com`
> - Triggering command: `/tmp/go-build2370360416/b264/config.test
/tmp/go-build2370360416/b264/config.test
-test.testlogfile=/tmp/go-build2370360416/b264/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/runtime/cgo
HEAD ache/go/1.25.6/x64/pkg/tool/linu-fmessage-length=0 get --global
rgo/bin/git 06.o -d 1.2.0/auth/auth.go git x_amd64/compile get --global
.12/x64/which x_amd64/compile` (dns block)
> - `nonexistent.local`
> - Triggering command: `/tmp/go-build2370360416/b279/launcher.test
/tmp/go-build2370360416/b279/launcher.test
-test.testlogfile=/tmp/go-build2370360416/b279/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/runtime/cgo
git ache/go/1.25.6/x64/pkg/tool/linu-o --global r it 08.o -d go git
x_amd64/vet
e/REDACTED/work/gh/opt/hostedtoolcache/go/1.25.6/x64/pkg/tool/linux_amd64/vet
64/src/runtime/c-unsafeptr=false 64/bin/git x_amd64/vet` (dns block)
> - `slow.example.com`
> - Triggering command: `/tmp/go-build2370360416/b279/launcher.test
/tmp/go-build2370360416/b279/launcher.test
-test.testlogfile=/tmp/go-build2370360416/b279/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/runtime/cgo
git ache/go/1.25.6/x64/pkg/tool/linu-o --global r it 08.o -d go git
x_amd64/vet
e/REDACTED/work/gh/opt/hostedtoolcache/go/1.25.6/x64/pkg/tool/linux_amd64/vet
64/src/runtime/c-unsafeptr=false 64/bin/git x_amd64/vet` (dns block)
> - `this-host-does-not-exist-12345.com`
> - Triggering command: `/tmp/go-build2370360416/b288/mcp.test
/tmp/go-build2370360416/b288/mcp.test
-test.testlogfile=/tmp/go-build2370360416/b288/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true g_.a
64/src/internal/bisect/bisect.go--64 86_64/as --global abis p/bin/git
ache/go/1.25.6/x/tmp/go-build2370360416/b059/vet.cfg -d ; then \
$GOPATH/bin/golangci-lint run --timeout=5m || echo &#34;��� Warning:
golangci-lint failed
/opt/hostedtoolcache/go/1.25.6/x64/pkg/tool/linux_amd64/link -wdcAJ1yv
x_amd64/vet rt-size &#39;1280,
7/opt/hostedtoolcache/go/1.25.6/x64/pkg/tool/linux_amd64/vet` (dns
block)
>
> If you need me to access, download, or install something from one of
these locations, you can either:
>
> - Configure [Actions setup
steps](https://gh.io/copilot/actions-setup-steps) to set up my
environment, which run before the firewall is enabled
> - Add the appropriate URLs or hosts to the custom allowlist in this
repository's [Copilot coding agent
settings](https://github.com/github/gh-aw-mcpg/settings/copilot/coding_agent)
(admins only)
>
> </details>

<!-- START COPILOT ORIGINAL PROMPT -->



<details>

<summary>Original prompt</summary>


----

*This section details on the original issue you should resolve*

<issue_title>📚 Documentation Reconciliation Report - February 11,
2026</issue_title>
<issue_description>## Summary

Found **1 critical** and **2 minor** discrepancies between documentation
and implementation during nightly reconciliation check.

- Workflow Run:
[§21890623217](https://github.com/github/gh-aw-mcpg/actions/runs/21890623217)
- Date: February 11, 2026
- Branch: main

## Critical Issues 🔴

Issues that would cause user confusion or broken workflows if followed:

### 1. Payload Size Threshold Default Value Mismatch

**Location:** README.md, lines 206, 281, 319

**Problem:** Documentation states the default payload size threshold is
`1024` bytes (1KB), but the actual implementation uses `10240` bytes
(10KB).

**Actual Behavior:**
- Code: `internal/cmd/flags_logging.go:14` defines
`defaultPayloadSizeThreshold = 10240`
- Code: `internal/config/config_payload.go:11` defines
`DefaultPayloadSizeThreshold = 10240`
- Tests: `internal/cmd/flags_logging_test.go` confirms default is 10240
bytes

**Impact:** Users expecting the documented 1KB threshold will actually
get a 10KB threshold, which could result in:
- More payloads being stored inline than expected (up to 10KB instead of
1KB)
- Less disk I/O than anticipated
- Different memory usage patterns
- Confusion when testing payload storage behavior

**Suggested Fix:** Update README.md to reflect the actual default:

```markdown
# Line 206
- CLI flag: `--payload-size-threshold (bytes)` (default: 10240)

# Line 281
--payload-size-threshold int   Size threshold (in bytes) for storing payloads to disk. Payloads larger than this are stored, smaller ones returned inline (default 10240)

# Line 319
| `MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD` | Size threshold in bytes for payload storage (sets default for `--payload-size-threshold` flag) | `10240` |
```

**Code Reference:**
- `internal/cmd/flags_logging.go:14`
- `internal/config/config_payload.go:11`
- `internal/cmd/flags_logging_test.go:46`

## Minor Issues 🔵

Small inconsistencies or missing details:

### 1. Internal Directory Structure Incomplete in CONTRIBUTING.md

**Location:** CONTRIBUTING.md, lines 206-233 (Project Structure section)

**Problem:** Documentation lists only 7 internal directories, but 16
actually exist.

**Missing Directories:**
- `internal/difc/` - Data Information Flow Control
- `internal/envutil/` - Environment variable utilities
- `internal/middleware/` - HTTP middleware (jq schema processing)
- `internal/sys/` - System utilities
- `internal/testutil/` - Test utilities and helpers
- `internal/tty/` - Terminal detection utilities
- `internal/version/` - Version management

**Impact:** Developers may be unaware of these internal packages when
working on the codebase, potentially duplicating functionality or
missing useful utilities.

**Suggested Fix:** Update the Project Structure section to include all
internal directories with brief descriptions.

**Code Reference:** Directory listing shows all 16 directories exist and
contain working code.

### 2. Internal Directory Structure Incomplete in AGENTS.md

**Location:** AGENTS.md, lines 12-20 (Project Structure section)

**Problem:** Similar to CONTRIBUTING.md, lists only 9 internal
directories instead of 16.

**Missing Directories:** Same as above (difc, envutil, middleware, sys,
testutil, tty, version)

**Impact:** AI agents working with the codebase may not be aware of all
available internal packages.

**Suggested Fix:** Update the Project Structure section to include all
internal directories.

**Code Reference:** Same directory listing as above.

## Documentation Completeness

### Accurate Sections ✅

The following sections were verified and found to be accurate:

- **Go Version Requirement** - Correctly documented as 1.25.0 (matches
`go.mod`)
- **Make Targets** - All 11 documented targets exist and work correctly:
  - `make build`, `test`, `test-unit`, `test-integration`, `test-all`
  - `make lint`, `coverage`, `install`, `agent-finished`
  - `make format`, `clean`
- **Configuration Fields** - All documented fields match code struct
definitions in `internal/config/`
- **Environment Variables** - Correctly documented and extensively used
(140+ references in code)
- **External Links**:
  - MCP Gateway spec link is valid (returns HTTP 200)
  - GitHub token URL is valid (redirects properly)
- **Feature Documentation**:
  - HTTP transport correctly marked as "fully supported"
- Container field requirement for stdio servers matches validation code
- Command field restriction for JSON stdin format accurately documented
  - Sequential launch flag exists and works as documented
- **CLI Flags** - All documented flags exist and match actual
implementation
- **Docker Configuration Examples** - Format and field names match
actual usage
- **Configuration Validation** - Documented behavior matches
`internal/config/validation*.go`

## Tested Commands

All commands from CONTRIBUTING....

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes #886

Author: lpcox

AGENTS.md

@@ -21,17 +21,24 @@ Quick reference for AI agents working with MCP Gateway (Go-based MCP proxy serve
 
 ## Project Structure
 
+- `internal/auth/` - Authentication header parsing and middleware
 - `internal/cmd/` - CLI (Cobra)
 - `internal/config/` - Config parsing (TOML/JSON) with validation
   - `validation.go` - Variable expansion and fail-fast validation
   - `validation_test.go` - 21 comprehensive validation tests
-- `internal/server/` - HTTP server (routed/unified modes)
-- `internal/mcp/` - MCP protocol types with enhanced error logging
-- `internal/launcher/` - Backend process management
+- `internal/difc/` - Data Information Flow Control
+- `internal/envutil/` - Environment variable utilities
 - `internal/guard/` - Security guards (NoopGuard active)
-- `internal/auth/` - Authentication header parsing and middleware
+- `internal/launcher/` - Backend process management
 - `internal/logger/` - Debug logging framework (micro logger)
+- `internal/mcp/` - MCP protocol types with enhanced error logging
+- `internal/middleware/` - HTTP middleware (jq schema processing)
+- `internal/server/` - HTTP server (routed/unified modes)
+- `internal/sys/` - System utilities
+- `internal/testutil/` - Test utilities and helpers
 - `internal/timeutil/` - Time formatting utilities
+- `internal/tty/` - Terminal detection utilities
+- `internal/version/` - Version management
 
 ## Key Tech
 

CONTRIBUTING.md

@@ -211,26 +211,42 @@ awmg/
 ├── Dockerfile                 # Container image
 ├── Makefile                   # Build automation
 └── internal/
+    ├── auth/                  # Authentication header parsing and middleware
     ├── cmd/                   # CLI commands (cobra)
     ├── config/                # Configuration loading (TOML/JSON)
+    ├── difc/                  # Data Information Flow Control
+    ├── envutil/               # Environment variable utilities
+    ├── guard/                 # Security guards (NoopGuard active)
     ├── launcher/              # Backend server management
+    ├── logger/                # Debug logging framework
     ├── mcp/                   # MCP protocol types & connection
+    ├── middleware/            # HTTP middleware (jq schema processing)
     ├── server/                # HTTP server (routed/unified modes)
-    ├── guard/                 # Security guards (NoopGuard active)
-    ├── logger/                # Debug logging framework
+    ├── sys/                   # System utilities
+    ├── testutil/              # Test utilities and helpers
     ├── timeutil/              # Time formatting utilities
-    └── tty/                   # Terminal detection utilities
+    ├── tty/                   # Terminal detection utilities
+    └── version/               # Version management
 ```
 
 ### Key Directories
 
+- **`internal/auth/`** - Authentication header parsing and middleware
 - **`internal/cmd/`** - CLI implementation using Cobra framework
 - **`internal/config/`** - Configuration parsing for TOML and JSON formats
-- **`internal/server/`** - HTTP server with routed and unified modes
-- **`internal/mcp/`** - MCP protocol types and JSON-RPC handling
-- **`internal/launcher/`** - Backend process management (Docker, stdio)
+- **`internal/difc/`** - Data Information Flow Control
+- **`internal/envutil/`** - Environment variable utilities
 - **`internal/guard/`** - Guard framework for resource labeling
+- **`internal/launcher/`** - Backend process management (Docker, stdio)
 - **`internal/logger/`** - Micro logger for debug output
+- **`internal/mcp/`** - MCP protocol types and JSON-RPC handling
+- **`internal/middleware/`** - HTTP middleware (jq schema processing)
+- **`internal/server/`** - HTTP server with routed and unified modes
+- **`internal/sys/`** - System utilities
+- **`internal/testutil/`** - Test utilities and helpers
+- **`internal/timeutil/`** - Time formatting utilities
+- **`internal/tty/`** - Terminal detection utilities
+- **`internal/version/`** - Version management
 
 ## Coding Conventions
 

README.md

@@ -202,7 +202,7 @@ See **[Configuration Specification](https://github.com/github/gh-aw/blob/main/do
 
 **Configuration Alternatives**:
 - **`payloadSizeThreshold`** is not supported in JSON stdin format. Use:
-  - CLI flag: `--payload-size-threshold <bytes>` (default: 1024)
+  - CLI flag: `--payload-size-threshold <bytes>` (default: 10240)
   - Environment variable: `MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD=<bytes>`
   - Payloads **larger** than this threshold are stored to disk and return metadata
   - Payloads **smaller than or equal** to this threshold are returned inline
@@ -278,7 +278,7 @@ Flags:
   -l, --listen string                HTTP server listen address (default "127.0.0.1:3000")
       --log-dir string               Directory for log files (falls back to stdout if directory cannot be created) (default "/tmp/gh-aw/mcp-logs")
       --payload-dir string           Directory for storing large payload files (segmented by session ID) (default "/tmp/jq-payloads")
-      --payload-size-threshold int   Size threshold (in bytes) for storing payloads to disk. Payloads larger than this are stored, smaller ones returned inline (default 1024)
+      --payload-size-threshold int   Size threshold (in bytes) for storing payloads to disk. Payloads larger than this are stored, smaller ones returned inline (default 10240)
       --routed                       Run in routed mode (each backend at /mcp/<server>)
       --sequential-launch   Launch MCP servers sequentially during startup (parallel launch is default)
       --unified             Run in unified mode (all backends at /mcp)
@@ -316,7 +316,7 @@ When running locally (`run.sh`), these variables are optional (warnings shown if
 | `MODE` | Gateway mode flag | `--routed` |
 | `MCP_GATEWAY_LOG_DIR` | Log file directory (sets default for `--log-dir` flag) | `/tmp/gh-aw/mcp-logs` |
 | `MCP_GATEWAY_PAYLOAD_DIR` | Large payload storage directory (sets default for `--payload-dir` flag) | `/tmp/jq-payloads` |
-| `MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD` | Size threshold in bytes for payload storage (sets default for `--payload-size-threshold` flag) | `1024` |
+| `MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD` | Size threshold in bytes for payload storage (sets default for `--payload-size-threshold` flag) | `10240` |
 
 ### Docker Configuration