Commit 71e75db
authored
Fix large payload middleware: populate CallToolResult.Content field (#766)
# Fix: Large Payload Middleware Not Returning Metadata to Clients
## Problem Statement
The MCP Gateway's large payload storage feature was not working as
expected. During test run 21762627709, the agent received a full 672KB
payload without any metadata (payloadPath, queryID, schema) despite the
middleware being enabled.
## Root Cause Analysis
### What Was Working ✅
1. Middleware was correctly applied to all non-sys tools
2. Payloads were being saved to
`/tmp/jq-payloads/{sessionID}/{queryID}/payload.json`
3. Metadata (queryID, payloadPath, schema) was being generated
4. Gateway was running in routed mode (default)
### What Was Broken ❌
The middleware returned the transformed response in the **second return
value**:
```go
return result, rewrittenResponse, nil
```
But when wrapping handlers for the MCP SDK, this value was
**discarded**:
```go
// In unified.go:363 and routed.go
result, _, err := handler(ctx, req, nil) // The _ throws away metadata!
return result, err
```
The MCP protocol only sends the `CallToolResult` to clients, which
wasn't being populated with the transformed response. Clients received
the original `CallToolResult` without any payload metadata.
## Solution Implemented
Modified `internal/middleware/jqschema.go` to:
1. Marshal the transformed response (queryID, payloadPath, schema,
preview, originalSize, truncated) to JSON
2. Create a **new** `CallToolResult` with the JSON in the `Content`
field as `TextContent`
3. Return this new result instead of the original
```go
transformedResult := &sdk.CallToolResult{
Content: []sdk.Content{
&sdk.TextContent{
Text: string(rewrittenJSON),
},
},
IsError: result.IsError,
Meta: result.Meta,
}
return transformedResult, rewrittenResponse, nil
```
## Enhanced Logging (Latest Update)
Added extensive logging throughout the payload storage process:
**Operational Logs (to file):**
- `logger.LogInfo` - Payload storage lifecycle, file operations, sizes
- `logger.LogDebug` - Directory creation, verification, metadata details
- `logger.LogWarn` - Schema generation failures, verification issues
- `logger.LogError` - Marshal failures, file write errors
**Debug Logs (console when DEBUG enabled):**
- `logMiddleware.Printf` - Existing debug output for development
**Key Logged Information:**
- Directory creation with full paths and permissions
- File write operations with size in bytes, KB, and MB
- Payload verification after write
- Metadata response generation
- Client-side path translation hints
- Error context with tool, queryID, and session information
## Testing & Verification
### Unit Tests ✅
- All 23 middleware tests pass
- Tests verify Content field contains metadata
- Tests verify truncation behavior
### Integration Tests ✅
- All 3 integration tests pass
- End-to-end flow verified
- Content field validation confirmed
### System Tests ✅
- All 28 integration tests pass
- Build completes successfully
- Linting passes
- Code review found no issues
- Security scan (CodeQL) found no vulnerabilities
## Impact & Benefits
### For Agents 🤖
- Will now receive metadata in tool responses
- Can access full payloads from mounted directory
- Can use schema to understand payload structure
### For Debugging 🔍
- Comprehensive logs track entire payload lifecycle
- File system operations fully visible
- Error conditions logged with context
- Easy to trace payload storage issues
### Backward Compatibility 🔄
- Minimal change to middleware only
- Internal data return value still contains transformed response
- Protocol-compliant using standard MCP TextContent
- No breaking changes to existing code
## Files Changed
- `internal/middleware/jqschema.go` - Core fix + enhanced logging
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
>
> ----
>
> *This section details on the original issue you should resolve*
>
> <issue_title>[large-payload-test] Large Payload Test -
21762627709</issue_title>
> <issue_description># Large MCP Payload Access Test Results
>
> **Run ID:** 21762627709
> **Status:** PARTIAL PASS
> **Timestamp:** 2026-02-06T19:10:21Z
>
> ## Test Results
>
> - **Expected Secret:**
test-secret-2786167c-221f-4a0f-a034-f88ef8394279
> - **Found Secret:** test-secret-2786167c-221f-4a0f-a034-f88ef8394279
> - **Secret Match:** YES ✅
> - **Payload Path:** NOT PROVIDED (feature not triggered)
> - **Payload Size:** 672,508 bytes (656.7 KB)
>
> ## Detailed Findings
>
> ### ✅ What Worked
> 1. Successfully read `secret.txt` from filesystem MCP server
> 2. Successfully read `large-test-file.json` (672KB file)
> 3. Secret validation passed - secrets match correctly
> 4. File content integrity verified
>
> ### 3 files changed
Lines changed: 202 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
116 | 120 | | |
| 121 | + | |
117 | 122 | | |
118 | 123 | | |
119 | 124 | | |
| 125 | + | |
| 126 | + | |
120 | 127 | | |
121 | 128 | | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
122 | 134 | | |
| 135 | + | |
| 136 | + | |
123 | 137 | | |
124 | 138 | | |
125 | 139 | | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
126 | 151 | | |
127 | 152 | | |
128 | 153 | | |
| |||
149 | 174 | | |
150 | 175 | | |
151 | 176 | | |
| 177 | + | |
| 178 | + | |
152 | 179 | | |
153 | 180 | | |
154 | 181 | | |
155 | 182 | | |
156 | 183 | | |
| 184 | + | |
| 185 | + | |
157 | 186 | | |
158 | 187 | | |
159 | 188 | | |
160 | 189 | | |
161 | 190 | | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
162 | 202 | | |
163 | 203 | | |
164 | 204 | | |
165 | 205 | | |
166 | 206 | | |
167 | 207 | | |
168 | 208 | | |
| 209 | + | |
| 210 | + | |
169 | 211 | | |
170 | 212 | | |
171 | 213 | | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
172 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
173 | 222 | | |
174 | 223 | | |
175 | 224 | | |
| 225 | + | |
| 226 | + | |
176 | 227 | | |
177 | 228 | | |
178 | 229 | | |
179 | 230 | | |
| 231 | + | |
| 232 | + | |
180 | 233 | | |
181 | 234 | | |
182 | 235 | | |
| 236 | + | |
183 | 237 | | |
184 | 238 | | |
185 | 239 | | |
| |||
196 | 250 | | |
197 | 251 | | |
198 | 252 | | |
| 253 | + | |
| 254 | + | |
199 | 255 | | |
200 | 256 | | |
201 | 257 | | |
202 | 258 | | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
203 | 262 | | |
204 | 263 | | |
205 | 264 | | |
206 | | - | |
| 265 | + | |
| 266 | + | |
207 | 267 | | |
| 268 | + | |
| 269 | + | |
208 | 270 | | |
209 | 271 | | |
| 272 | + | |
| 273 | + | |
210 | 274 | | |
211 | 275 | | |
212 | 276 | | |
| |||
216 | 280 | | |
217 | 281 | | |
218 | 282 | | |
219 | | - | |
| 283 | + | |
220 | 284 | | |
221 | 285 | | |
222 | 286 | | |
223 | | - | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
224 | 290 | | |
225 | 291 | | |
226 | 292 | | |
227 | 293 | | |
228 | 294 | | |
229 | 295 | | |
230 | 296 | | |
231 | | - | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
232 | 329 | | |
233 | 330 | | |
234 | 331 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
68 | 91 | | |
69 | 92 | | |
70 | 93 | | |
| |||
180 | 203 | | |
181 | 204 | | |
182 | 205 | | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
183 | 225 | | |
184 | 226 | | |
185 | 227 | | |
| |||
223 | 265 | | |
224 | 266 | | |
225 | 267 | | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
226 | 280 | | |
227 | 281 | | |
228 | 282 | | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
229 | 286 | | |
230 | 287 | | |
231 | 288 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
153 | 161 | | |
154 | 162 | | |
155 | | - | |
| 163 | + | |
156 | 164 | | |
157 | 165 | | |
158 | 166 | | |
159 | 167 | | |
160 | | - | |
| 168 | + | |
161 | 169 | | |
162 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
163 | 177 | | |
164 | 178 | | |
165 | 179 | | |
| |||
216 | 230 | | |
217 | 231 | | |
218 | 232 | | |
219 | | - | |
220 | | - | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
221 | 242 | | |
222 | | - | |
223 | | - | |
224 | | - | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
225 | 246 | | |
226 | 247 | | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
227 | 253 | | |
228 | 254 | | |
229 | 255 | | |
| |||
0 commit comments