Skip to content

Commit 5c071a7

Browse files
dannykoppingdannykopping
authored
feat: record model thoughts (#203)
* feat: record model thoughts Signed-off-by: Danny Kopping <danny@coder.com> * fix: send model thoughts with tool usage recording Signed-off-by: Danny Kopping <danny@coder.com> * feat: capture responses reasoning Signed-off-by: Danny Kopping <danny@coder.com> * chore: refactor tests Signed-off-by: Danny Kopping <danny@coder.com> * chore: test multiple thoughts Signed-off-by: Danny Kopping <danny@coder.com> * chore: refactor tests Signed-off-by: Danny Kopping <danny@coder.com> * feat: capture commentary as model thoughts Signed-off-by: Danny Kopping <danny@coder.com> * chore: cleaning up /v1/messages impl Signed-off-by: Danny Kopping <danny@coder.com> * chore: update comments Signed-off-by: Danny Kopping <danny@coder.com> * chore: add tests for parallel tool calls Signed-off-by: Danny Kopping <danny@coder.com> * chore: capture source of thinking/reasoning Signed-off-by: Danny Kopping <danny@coder.com> * chore: break association between thoughts and tools Signed-off-by: Danny Kopping <danny@coder.com> * chore: expand trace testing Signed-off-by: Danny Kopping <danny@coder.com> * chore: refactor model thought assertions into common func Signed-off-by: Danny Kopping <danny@coder.com> --------- Signed-off-by: Danny Kopping <danny@coder.com>
1 parent b3cd38b commit 5c071a7

30 files changed

Lines changed: 1899 additions & 56 deletions

‎api.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ type (
3030
TokenUsageRecord = recorder.TokenUsageRecord
3131
PromptUsageRecord = recorder.PromptUsageRecord
3232
ToolUsageRecord = recorder.ToolUsageRecord
33+
ModelThoughtRecord = recorder.ModelThoughtRecord
3334
Recorder = recorder.Recorder
3435
Metadata = recorder.Metadata
3536

‎fixtures/anthropic/multi_thinking_builtin_tool.txtar‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
Claude Code has builtin tools to (e.g.) explore the filesystem.
2+
This fixture has two thinking blocks before the tool_use block.
3+
4+
-- request --
5+
{
6+
"model": "claude-sonnet-4-20250514",
7+
"max_tokens": 1024,
8+
"tools": [
9+
{
10+
"name": "Read",
11+
"description": "Read the contents of a file at the given path.",
12+
"input_schema": {
13+
"type": "object",
14+
"properties": {
15+
"file_path": {
16+
"type": "string",
17+
"description": "The absolute path to the file to read"
18+
}
19+
},
20+
"required": ["file_path"]
21+
}
22+
}
23+
],
24+
"messages": [
25+
{
26+
"role": "user",
27+
"content": "read the foo file"
28+
}
29+
]
30+
}
31+
32+
-- streaming --
33+
event: message_start
34+
data: {"type":"message_start","message":{"id":"msg_015SQewixvT9s4cABCVvUE6g","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":22,"cache_read_input_tokens":13993,"output_tokens":5,"service_tier":"standard"}} }
35+
36+
event: content_block_start
37+
data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}
38+
39+
event: content_block_delta
40+
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"The user wants me to read a file called \"foo\". Let me find and read it."}}
41+
42+
event: content_block_delta
43+
data: {"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"Eu8BCkYICxgCKkBR++kFr7Za2JhF/9OCpjEc46/EcipL75RK+MEbxJ/VBJPWQTWrNGfwb5khWYJtKEpjjkH07cR/MQvThfb7t7CkEgwU4pKwL7NuZXd1/wgaDILyd0bYMqQovWo3dyIw95Ny7yZPljNBDLsvMBdBr7w+RtbU+AlSftjBuBZHp0VzI54/W+9u6f7qfx0JXsVBKldqqOjFvewT8Xm6Qp/77g6/j0zBiuAQABj/6vS1qATjd8KSIFDg9G/tCtzwmV/T/egmzswWd5CBiAhW6lgJgEDRr+gRUrFSOB7o3hypW8FUnUrr1JtzzwMYAQ=="}}
44+
45+
event: content_block_stop
46+
data: {"type":"content_block_stop","index":0}
47+
48+
event: content_block_start
49+
data: {"type":"content_block_start","index":1,"content_block":{"type":"thinking","thinking":""}}
50+
51+
event: content_block_delta
52+
data: {"type":"content_block_delta","index":1,"delta":{"type":"thinking_delta","thinking":"I should use the Read tool to access the file contents."}}
53+
54+
event: content_block_delta
55+
data: {"type":"content_block_delta","index":1,"delta":{"type":"signature_delta","signature":"Aa1BCkYICxgCKkBR++kFr7Za2JhF/9OCpjEc46/EcipL75RK+MEbxJ/VBJPWQTWrNGfwb5khWYJtKEpjjkH07cR/MQvThfb7t7CkEgwU4pKwL7NuZXd1/wgaDILyd0bYMqQovWo3dyIw95Ny7yZPljNBDLsvMBdBr7w+RtbU+AlSftjBuBZHp0VzI54/W+9u6f7qfx0JXsVBKldqqOjFvewT8Xm6Qp/77g6/j0zBiuAQABj/6vS1qATjd8KSIFDg9G/tCtzwmV/T/egmzswWd5CBiAhW6lgJgEDRr+gRUrFSOB7o3hypW8FUnUrr1JtzzwMYAQ=="}}
56+
57+
event: content_block_stop
58+
data: {"type":"content_block_stop","index":1}
59+
60+
event: content_block_start
61+
data: {"type":"content_block_start","index":2,"content_block":{"type":"tool_use","id":"toolu_01RX68weRSquLx6HUTj65iBo","name":"Read","input":{}}}
62+
63+
event: ping
64+
data: {"type": "ping"}
65+
66+
event: content_block_delta
67+
data: {"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":""} }
68+
69+
event: content_block_delta
70+
data: {"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":"{\"file_path\": \"/tmp/blah/foo"} }
71+
72+
event: content_block_delta
73+
data: {"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":"\"}"} }
74+
75+
event: content_block_stop
76+
data: {"type":"content_block_stop","index":2 }
77+
78+
event: message_delta
79+
data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":61} }
80+
81+
event: message_stop
82+
data: {"type":"message_stop" }
83+
84+
85+
-- non-streaming --
86+
{
87+
"id": "msg_01JHKqEmh7wYuPXqUWUvusfL",
88+
"container": {
89+
"id": "",
90+
"expires_at": "0001-01-01T00:00:00Z"
91+
},
92+
"content": [
93+
{
94+
"type": "thinking",
95+
"thinking": "The user wants me to read a file called \"foo\". Let me find and read it.",
96+
"signature": "Eu8BCkYICxgCKkBR++kFr7Za2JhF/9OCpjEc46/EcipL75RK+MEbxJ/VBJPWQTWrNGfwb5khWYJtKEpjjkH07cR/MQvThfb7t7CkEgwU4pKwL7NuZXd1/wgaDILyd0bYMqQovWo3dyIw95Ny7yZPljNBDLsvMBdBr7w+RtbU+AlSftjBuBZHp0VzI54/W+9u6f7qfx0JXsVBKldqqOjFvewT8Xm6Qp/77g6/j0zBiuAQABj/6vS1qATjd8KSIFDg9G/tCtzwmV/T/egmzswWd5CBiAhW6lgJgEDRr+gRUrFSOB7o3hypW8FUnUrr1JtzzwMYAQ=="
97+
},
98+
{
99+
"type": "thinking",
100+
"thinking": "I should use the Read tool to access the file contents.",
101+
"signature": "Aa1BCkYICxgCKkBR++kFr7Za2JhF/9OCpjEc46/EcipL75RK+MEbxJ/VBJPWQTWrNGfwb5khWYJtKEpjjkH07cR/MQvThfb7t7CkEgwU4pKwL7NuZXd1/wgaDILyd0bYMqQovWo3dyIw95Ny7yZPljNBDLsvMBdBr7w+RtbU+AlSftjBuBZHp0VzI54/W+9u6f7qfx0JXsVBKldqqOjFvewT8Xm6Qp/77g6/j0zBiuAQABj/6vS1qATjd8KSIFDg9G/tCtzwmV/T/egmzswWd5CBiAhW6lgJgEDRr+gRUrFSOB7o3hypW8FUnUrr1JtzzwMYAQ=="
102+
},
103+
{
104+
"citations": null,
105+
"text": "",
106+
"type": "tool_use",
107+
"id": "toolu_01AusGgY5aKFhzWrFBv9JfHq",
108+
"input": {
109+
"file_path": "/tmp/blah/foo"
110+
},
111+
"name": "Read",
112+
"content": {
113+
"OfWebSearchResultBlockArray": null,
114+
"OfString": "",
115+
"OfMCPToolResultBlockContent": null,
116+
"error_code": "",
117+
"type": "",
118+
"content": null,
119+
"return_code": 0,
120+
"stderr": "",
121+
"stdout": ""
122+
},
123+
"tool_use_id": "",
124+
"server_name": "",
125+
"is_error": false,
126+
"file_id": "",
127+
"signature": "",
128+
"thinking": "",
129+
"data": ""
130+
}
131+
],
132+
"model": "claude-sonnet-4-20250514",
133+
"role": "assistant",
134+
"stop_reason": "tool_use",
135+
"stop_sequence": "",
136+
"type": "message",
137+
"usage": {
138+
"cache_creation": {
139+
"ephemeral_1h_input_tokens": 0,
140+
"ephemeral_5m_input_tokens": 0
141+
},
142+
"cache_creation_input_tokens": 0,
143+
"cache_read_input_tokens": 23490,
144+
"input_tokens": 5,
145+
"output_tokens": 84,
146+
"server_tool_use": {
147+
"web_search_requests": 0
148+
},
149+
"service_tier": "standard"
150+
}
151+
}
152+

‎fixtures/anthropic/simple.txtar‎

Lines changed: 41 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -23,91 +23,100 @@ event: message_start
2323
data: {"type":"message_start","message":{"id":"msg_01Pvyf26bY17RcjmWfJsXGBn","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":18,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":1,"service_tier":"standard"}} }
2424

2525
event: content_block_start
26-
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} }
26+
data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}
27+
28+
event: content_block_delta
29+
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"This is a classic philosophical question about medieval scholasticism. I'll give a thoughtful answer."}}
30+
31+
event: content_block_stop
32+
data: {"type":"content_block_stop","index":0}
33+
34+
event: content_block_start
35+
data: {"type":"content_block_start","index":1,"content_block":{"type":"text","text":""} }
2736

2837
event: ping
2938
data: {"type": "ping"}
3039

3140
event: content_block_delta
32-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"This"} }
41+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"This"} }
3342

3443
event: content_block_delta
35-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" is a famous philosophical question often used to illustrate medieval"}}
44+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" is a famous philosophical question often used to illustrate medieval"}}
3645

3746
event: content_block_delta
38-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" scholastic debates that seem pointless or ov"} }
47+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" scholastic debates that seem pointless or ov"} }
3948

4049
event: content_block_delta
41-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"erly abstract. The question \"How many angels can dance on the head of"} }
50+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"erly abstract. The question \"How many angels can dance on the head of"} }
4251

4352
event: content_block_delta
44-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" a pin?\" is typically cited as an example of us"} }
53+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" a pin?\" is typically cited as an example of us"} }
4554

4655
event: content_block_delta
47-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"eless speculation.\n\nHistorically, medieval theolog"} }
56+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"eless speculation.\n\nHistorically, medieval theolog"} }
4857

4958
event: content_block_delta
50-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ians did debate the nature of angels -"} }
59+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"ians did debate the nature of angels -"} }
5160

5261
event: content_block_delta
53-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" whether they were incorporeal beings, how"}}
62+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" whether they were incorporeal beings, how"}}
5463

5564
event: content_block_delta
56-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" they occupied space, and whether multiple angels could exist"} }
65+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" they occupied space, and whether multiple angels could exist"} }
5766

5867
event: content_block_delta
59-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" in the same location. However, there"} }
68+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" in the same location. However, there"} }
6069

6170
event: content_block_delta
62-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"'s little evidence they literally"} }
71+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"'s little evidence they literally"} }
6372

6473
event: content_block_delta
65-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" debated dancing angels on pinheads.\n\nThe question has"} }
74+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" debated dancing angels on pinheads.\n\nThe question has"} }
6675

6776
event: content_block_delta
68-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" no factual answer since it depends on assumptions about:"}}
77+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" no factual answer since it depends on assumptions about:"}}
6978

7079
event: content_block_delta
71-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"\n- The existence and nature of angels\n- Whether"} }
80+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"\n- The existence and nature of angels\n- Whether"} }
7281

7382
event: content_block_delta
74-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" incorporeal beings occupy physical space\n- What"} }
83+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" incorporeal beings occupy physical space\n- What"} }
7584

7685
event: content_block_delta
77-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" constitutes \"dancing\" for a spiritual"} }
86+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" constitutes \"dancing\" for a spiritual"} }
7887

7988
event: content_block_delta
80-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" entity\n- The size of both the"} }
89+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" entity\n- The size of both the"} }
8190

8291
event: content_block_delta
83-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" pin and the angels\n\nIt's become a metaph"} }
92+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" pin and the angels\n\nIt's become a metaph"} }
8493

8594
event: content_block_delta
86-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"or for overthinking trivial matters"} }
95+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"or for overthinking trivial matters"} }
8796

8897
event: content_block_delta
89-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" or getting lost in theoretical discussions disconnected from practical reality."} }
98+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" or getting lost in theoretical discussions disconnected from practical reality."} }
9099

91100
event: content_block_delta
92-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" Some use it to critique certain types of academic"} }
101+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" Some use it to critique certain types of academic"} }
93102

94103
event: content_block_delta
95-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" or theological debate, while others defen"} }
104+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" or theological debate, while others defen"} }
96105

97106
event: content_block_delta
98-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"d the value of exploring fundamental questions about existence an"} }
107+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"d the value of exploring fundamental questions about existence an"} }
99108

100109
event: content_block_delta
101-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"d metaphysics.\n\nSo while u"} }
110+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"d metaphysics.\n\nSo while u"} }
102111

103112
event: content_block_delta
104-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"nanswerable literally, it serves as an interesting lens"} }
113+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"nanswerable literally, it serves as an interesting lens"} }
105114

106115
event: content_block_delta
107-
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" for discussing the nature of philosophical inquiry itself."} }
116+
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" for discussing the nature of philosophical inquiry itself."} }
108117

109118
event: content_block_stop
110-
data: {"type":"content_block_stop","index":0 }
119+
data: {"type":"content_block_stop","index":1 }
111120

112121
event: message_delta
113122
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":240} }
@@ -122,6 +131,10 @@ data: {"type":"message_stop" }
122131
"role": "assistant",
123132
"model": "claude-sonnet-4-20250514",
124133
"content": [
134+
{
135+
"type": "thinking",
136+
"thinking": "This is a classic philosophical question about medieval scholasticism. I'll give a thoughtful answer."
137+
},
125138
{
126139
"type": "text",
127140
"text": "This is a famous philosophical question, often called \"How many angels can dance on the head of a pin?\" It's typically used to represent pointless or overly abstract theological debates.\n\nThe question doesn't have a literal answer because:\n\n1. **Historical context**: It's often attributed to medieval scholastic philosophers, though there's little evidence they actually debated this exact question. It became a popular way to mock what some saw as useless academic arguments.\n\n2. **Philosophical purpose**: The question highlights the difficulty of discussing non-physical beings (angels) in physical terms (space on a pinhead).\n\n3. **Different interpretations**: \n - If angels are purely spiritual, they might not take up physical space at all\n - If they do occupy space, we'd need to know their \"size\"\n - The question might be asking about the nature of space, matter, and spirit\n\nSo the real answer is that it's not meant to be answered literally - it's a thought experiment about the limits of rational inquiry and the sometimes absurd directions theological speculation can take.\n\nWould you like to explore the philosophical implications behind this question, or were you thinking about it in a different context?"

0 commit comments

Comments
 (0)