Skip to content

Commit be073f2

Browse files
SteveSandersonMSCopilot
SteveSandersonMS
and
Copilot
committed
Fix proxy normalization for array-content user messages
The replaying proxy's message normalization only handled string content for user messages. Array content (multimodal messages with image_url parts) was silently dropped, producing blank user entries that couldn't match incoming requests containing image content. Extract text parts and represent image_url parts as [image] markers so both capture and replay sides normalize consistently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 21f0661 commit be073f2

File tree

3 files changed

+123
-38
lines changed

3 files changed

+123
-38
lines changed

‎test/harness/replayingCapiProxy.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -721,6 +721,18 @@ function transformOpenAIRequestMessage(
721721
content = "${system}";
722722
} else if (m.role === "user" && typeof m.content === "string") {
723723
content = normalizeUserMessage(m.content);
724+
} else if (m.role === "user" && Array.isArray(m.content)) {
725+
// Multimodal user messages have array content with text and image_url parts.
726+
// Extract and normalize text parts; represent image_url parts as a stable marker.
727+
const parts: string[] = [];
728+
for (const part of m.content) {
729+
if (typeof part === "object" && part.type === "text" && typeof part.text === "string") {
730+
parts.push(normalizeUserMessage(part.text));
731+
} else if (typeof part === "object" && part.type === "image_url") {
732+
parts.push("[image]");
733+
}
734+
}
735+
content = parts.join("\n") || undefined;
724736
} else if (m.role === "tool" && typeof m.content === "string") {
725737
// If it's a JSON tool call result, normalize the whitespace and property ordering.
726738
// For successful tool results wrapped in {resultType, textResultForLlm}, unwrap to

‎test/snapshots/session_config/vision_disabled_then_enabled_via_setmodel.yaml‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -44,9 +44,9 @@ conversations:
4444
tool_call_id: toolcall_1
4545
content: Viewed image file successfully.
4646
- role: assistant
47-
content: I can see the image shows **GitHub Copilot** branding. The image contains the GitHub Copilot logo (a hexagonal
48-
icon with colorful segments) along with text that reads "GitHub Copilot" in a clean, modern font. The design
49-
appears to be on a dark background, typical of GitHub's branding materials.
47+
content: The image shows a **GitHub Copilot logo** - a hexagonal icon with an orange/coral colored background containing
48+
a stylized white design that represents the Copilot brand mark. The logo has the characteristic GitHub Copilot
49+
appearance with its geometric hexagon shape and minimal design aesthetic.
5050
- role: user
5151
content: Use the view tool to look at the file test.png and describe what you see
5252
- role: assistant
@@ -87,9 +87,9 @@ conversations:
8787
tool_call_id: toolcall_1
8888
content: Viewed image file successfully.
8989
- role: assistant
90-
content: I can see the image shows **GitHub Copilot** branding. The image contains the GitHub Copilot logo (a hexagonal
91-
icon with colorful segments) along with text that reads "GitHub Copilot" in a clean, modern font. The design
92-
appears to be on a dark background, typical of GitHub's branding materials.
90+
content: The image shows a **GitHub Copilot logo** - a hexagonal icon with an orange/coral colored background containing
91+
a stylized white design that represents the Copilot brand mark. The logo has the characteristic GitHub Copilot
92+
appearance with its geometric hexagon shape and minimal design aesthetic.
9393
- role: user
9494
content: Use the view tool to look at the file test.png and describe what you see
9595
- role: assistant
@@ -111,8 +111,10 @@ conversations:
111111
tool_call_id: toolcall_3
112112
content: Viewed image file successfully.
113113
- role: user
114+
content: |-
115+
Image file at path ${workdir}/test.png
116+
[image]
114117
- role: assistant
115-
content: Looking at the image, I can see a very simple, minimalist graphic. It appears to be a small, pale rectangular
116-
shape (possibly light beige or cream colored) on a white background. The rectangle looks like it might be a
117-
simple placeholder, button, or UI element. The image is quite small and doesn't contain any text, icons, or
118-
complex graphics - just this basic rectangular form.
118+
content: The image appears to be a **very small, mostly blank white or off-white rectangular image**. There doesn't seem
119+
to be any visible content, text, or graphics - just a plain light-colored background. It looks like either an
120+
empty/blank image file or a very minimalistic placeholder image.

‎test/snapshots/session_config/vision_enabled_then_disabled_via_setmodel.yaml‎

Lines changed: 99 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,17 @@ conversations:
77
- role: user
88
content: Use the view tool to look at the file test.png and describe what you see
99
- role: assistant
10-
content: I'll view the test.png file for you.
10+
content: I'll use the view tool to examine the test.png file.
1111
- role: assistant
1212
tool_calls:
1313
- id: toolcall_0
14+
type: function
15+
function:
16+
name: report_intent
17+
arguments: '{"intent":"Viewing image file"}'
18+
- role: assistant
19+
tool_calls:
20+
- id: toolcall_1
1421
type: function
1522
function:
1623
name: view
@@ -21,76 +28,140 @@ conversations:
2128
- role: user
2229
content: Use the view tool to look at the file test.png and describe what you see
2330
- role: assistant
24-
content: I'll view the test.png file for you.
31+
content: I'll use the view tool to examine the test.png file.
2532
tool_calls:
2633
- id: toolcall_0
34+
type: function
35+
function:
36+
name: report_intent
37+
arguments: '{"intent":"Viewing image file"}'
38+
- id: toolcall_1
2739
type: function
2840
function:
2941
name: view
3042
arguments: '{"path":"${workdir}/test.png"}'
3143
- role: tool
3244
tool_call_id: toolcall_0
45+
content: Intent logged
46+
- role: tool
47+
tool_call_id: toolcall_1
3348
content: Viewed image file successfully.
3449
- role: user
50+
content: |-
51+
Image file at path ${workdir}/test.png
52+
[image]
3553
- role: assistant
3654
content: >-
37-
The image appears to be almost entirely blank or white. There's a very subtle, light yellowish or
38-
cream-colored tint, but no discernible text, graphics, or other visual elements are visible. It looks like
39-
either:
40-
41-
42-
1. A blank/empty image file
55+
Looking at the test.png image, I can see what appears to be a solid beige or tan colored square. It's a very
56+
simple image - just a single uniform color with no visible patterns, text, or other elements. The color is a
57+
light peachy-tan shade, similar to a skin tone or sand color.
4358
44-
2. An extremely faded or overexposed image
4559
46-
3. A placeholder image with just a very light background color
47-
48-
49-
The image doesn't contain any meaningful visual content that I can detect.
60+
This looks like it could be a test image or placeholder, possibly just a single pixel or small solid color
61+
block that's been created for testing purposes.
5062
- messages:
5163
- role: system
5264
content: ${system}
5365
- role: user
5466
content: Use the view tool to look at the file test.png and describe what you see
5567
- role: assistant
56-
content: I'll view the test.png file for you.
68+
content: I'll use the view tool to examine the test.png file.
5769
tool_calls:
5870
- id: toolcall_0
71+
type: function
72+
function:
73+
name: report_intent
74+
arguments: '{"intent":"Viewing image file"}'
75+
- id: toolcall_1
5976
type: function
6077
function:
6178
name: view
6279
arguments: '{"path":"${workdir}/test.png"}'
6380
- role: tool
6481
tool_call_id: toolcall_0
82+
content: Intent logged
83+
- role: tool
84+
tool_call_id: toolcall_1
6585
content: Viewed image file successfully.
6686
- role: assistant
6787
content: >-
68-
The image appears to be almost entirely blank or white. There's a very subtle, light yellowish or
69-
cream-colored tint, but no discernible text, graphics, or other visual elements are visible. It looks like
70-
either:
71-
72-
73-
1. A blank/empty image file
74-
75-
2. An extremely faded or overexposed image
76-
77-
3. A placeholder image with just a very light background color
88+
Looking at the test.png image, I can see what appears to be a solid beige or tan colored square. It's a very
89+
simple image - just a single uniform color with no visible patterns, text, or other elements. The color is a
90+
light peachy-tan shade, similar to a skin tone or sand color.
7891
7992
80-
The image doesn't contain any meaningful visual content that I can detect.
93+
This looks like it could be a test image or placeholder, possibly just a single pixel or small solid color
94+
block that's been created for testing purposes.
95+
- role: user
96+
content: Use the view tool to look at the file test.png and describe what you see
97+
- role: assistant
98+
tool_calls:
99+
- id: toolcall_2
100+
type: function
101+
function:
102+
name: report_intent
103+
arguments: '{"intent":"Viewing image file"}'
104+
- role: assistant
105+
tool_calls:
106+
- id: toolcall_3
107+
type: function
108+
function:
109+
name: view
110+
arguments: '{"path":"${workdir}/test.png"}'
111+
- messages:
112+
- role: system
113+
content: ${system}
81114
- role: user
82115
content: Use the view tool to look at the file test.png and describe what you see
83116
- role: assistant
117+
content: I'll use the view tool to examine the test.png file.
84118
tool_calls:
119+
- id: toolcall_0
120+
type: function
121+
function:
122+
name: report_intent
123+
arguments: '{"intent":"Viewing image file"}'
85124
- id: toolcall_1
86125
type: function
87126
function:
88127
name: view
89128
arguments: '{"path":"${workdir}/test.png"}'
129+
- role: tool
130+
tool_call_id: toolcall_0
131+
content: Intent logged
90132
- role: tool
91133
tool_call_id: toolcall_1
92134
content: Viewed image file successfully.
93135
- role: assistant
94-
content: The image shows what appears to be a very light, nearly white background with an extremely subtle yellowish or
95-
cream tint. There are no visible text elements, shapes, graphics, or other discernible content in the image.
96-
It's essentially a blank or nearly blank image file.
136+
content: >-
137+
Looking at the test.png image, I can see what appears to be a solid beige or tan colored square. It's a very
138+
simple image - just a single uniform color with no visible patterns, text, or other elements. The color is a
139+
light peachy-tan shade, similar to a skin tone or sand color.
140+
141+
142+
This looks like it could be a test image or placeholder, possibly just a single pixel or small solid color
143+
block that's been created for testing purposes.
144+
- role: user
145+
content: Use the view tool to look at the file test.png and describe what you see
146+
- role: assistant
147+
tool_calls:
148+
- id: toolcall_2
149+
type: function
150+
function:
151+
name: report_intent
152+
arguments: '{"intent":"Viewing image file"}'
153+
- id: toolcall_3
154+
type: function
155+
function:
156+
name: view
157+
arguments: '{"path":"${workdir}/test.png"}'
158+
- role: tool
159+
tool_call_id: toolcall_2
160+
content: Intent logged
161+
- role: tool
162+
tool_call_id: toolcall_3
163+
content: Viewed image file successfully.
164+
- role: assistant
165+
content: I can see the test.png image again. It shows a solid, uniform beige or tan colored square - a simple test image
166+
with just one flat color throughout. The color is a light peachy-tan or sand-like shade. There's no text,
167+
patterns, gradients, or other visual elements - just a single solid color filling the entire image.

0 commit comments

Comments
 (0)