You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Tool calls now live on the conversation model, so the existing agent
evaluators run across a whole conversation, not just a single agent
turn.
An assistant `Message` carries the tool calls it made (a typed
`List<ToolCall>`, the same model the agent evaluators already use).
`ConversationTrajectory` exposes them flat (`toolCalls()`) and per turn
(`toolCallsByTurn()`), and bridges to the evaluators with
`toAgentOutputs()` and `toTestCase(...)`. The deterministic tool
evaluators and the judge ones run unchanged over the conversation.
## What's included
- `Message` gains a typed `toolCalls` component and `assistant(content,
toolCalls)`. The 3-arg constructor stays, so existing call sites compile
and link unchanged.
- `ConversationTrajectory`: `toolCalls()`, `toolCallsByTurn()`,
`toAgentTrace()`, `toAgentOutputs()`, and `toTestCase()`/`(tools)`
(deterministic, last user message as input) versus `toTestCase(tools,
tasks)` (judge path, full transcript as input so the dialog is not
re-wrapped).
- `TrajectoryEvaluator.includeToolCalls(boolean)`, default off, so
existing judge prompts are byte-identical.
- `dev.dokimos.core.agents.ToolCalls.coerce(...)`, shared by the bridge
and `AgentEvalCasts`.
- Kotlin DSL overloads, a runnable `MultiTurnToolCallExample`, docs, and
a live `LangChain4jMultiTurnToolIT`.
Reuses the existing `ToolCall`, so a tool call on a turn keeps its typed
`resultAs`/`argumentsAs`/`resultJson`. Tool-free conversations serialize
byte-identically; `toText()`/`toJson()` only change when a turn has
calls.
Copy file name to clipboardExpand all lines: docs/docs/evaluation/agent-evaluation.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -419,6 +419,10 @@ val withTools = trace.toTestCase("Find flights from NYC to Paris", tools)
419
419
</TabItem>
420
420
</Tabs>
421
421
422
+
:::tip Multi-turn agents
423
+
These evaluators score one set of tool calls. When tools are called across a back-and-forth conversation, attach the calls to each assistant turn and score the conversation per turn, with the same evaluators and no LLM. A `ConversationTrajectory` exposes `toolCallsByTurn()` for per-turn scoring and `toTestCase(tools)` / `toTestCase(tools, tasks)` for the whole-conversation deterministic and judge paths. See [Tool Calls on Turns](./multi-turn-conversations.md#tool-calls-on-turns).
424
+
:::
425
+
422
426
## Extracting Traces from Your Framework
423
427
424
428
The examples above assume you already have an `AgentTrace`. In practice your agent runs on a framework, and Dokimos ships extractors that turn a framework's own run result into an `AgentTrace`, so you don't hand-write the mapping. Each extractor captures the tool calls (name, parsed arguments, and result) and the final response.
Copy file name to clipboardExpand all lines: docs/docs/evaluation/multi-turn-conversations.md
+184Lines changed: 184 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -174,6 +174,190 @@ trajectory.toText() // Plain text transcript
174
174
</TabItem>
175
175
</Tabs>
176
176
177
+
### Tool Calls on Turns
178
+
179
+
A real agent calls tools mid-conversation: it looks up the weather, searches flights, then books a hotel. An assistant turn can carry the tool calls it made, so you can score *what the agent did each turn*, not just what it said.
180
+
181
+
Attach a typed `List<ToolCall>` to an assistant turn. A turn that called no tools needs no change.
ToolCall.of("book_hotel", mapOf("city" to "Paris"))
213
+
))
214
+
user("Thanks!")
215
+
assistant("You're all set!") // tool-free turn, unchanged
216
+
}
217
+
```
218
+
219
+
</TabItem>
220
+
</Tabs>
221
+
222
+
`Message` carries the tool calls as a typed `List<ToolCall>`; an assistant message built without them returns an empty list. When your app produces a turn, attach the calls with `Message.assistant(content, toolCalls)`.
223
+
224
+
#### Per-Turn Evaluation (Primary Path)
225
+
226
+
This is the recommended way to grade tool use across a conversation. `toolCallsByTurn()` returns one tool-call list per assistant turn, in order. Pair each turn with the calls you expected and run the [deterministic agent evaluators](./agent-evaluation.md), with no LLM and no API key.
`toolCallsByTurn()` groups by **assistant message**, which can differ from `turnCount()` (user/assistant pairs) when a conversation has consecutive or leading assistant messages. Each inner list lines up with `assistantMessages()`.
285
+
:::
286
+
287
+
See [`MultiTurnToolCallExample.java`](https://github.com/dokimos-dev/dokimos/blob/master/dokimos-examples/src/main/java/dev/dokimos/examples/conversation/MultiTurnToolCallExample.java) for a complete runnable version.
288
+
289
+
#### Whole-Conversation Shortcuts
290
+
291
+
When you want to assert over the whole conversation rather than per turn, build a test case straight from the trajectory.
292
+
293
+
-`toolCalls()`: every turn's calls flattened into one list, in order.
294
+
-`toTestCase()` and `toTestCase(tools)`: a **deterministic** test case. The flattened `toolCalls` go in the actual outputs, the input is the **last user message**, and `tools` (when given) go in metadata. As-is, it feeds the rule-based evaluators that read only actual outputs (validity, error, efficiency). `ToolCorrectnessEvaluator` and `ToolTrajectoryEvaluator` additionally need an expected list, which this path does not set; wire one in yourself (for example, `EvalTestCase.builder().expectedOutput("toolCalls", expected)`) or they throw an `EvaluationException`.
295
+
-`toTestCase(tools, tasks)`: the **judge** test case for `TaskCompletionEvaluator` and `ToolArgumentHallucinationEvaluator`. Its input is the rendered transcript of the whole conversation, but tool calls are rendered **name-only** (`[tool: name]`, not `[tool: name(args)]`) so the argument values a hallucination judge assesses never appear in the grounding it reads; the arguments stay available through the actual outputs. No separate output is set, so the transcript is not double-wrapped.
296
+
-`toAgentTrace()` / `toAgentOutputs()`: collapse the conversation into a single `AgentTrace` (or its output map) for the standard agent data flow.
297
+
298
+
<TabsgroupId="lang"defaultValue="java">
299
+
<TabItemvalue="java"label="Java">
300
+
301
+
```java
302
+
// Deterministic: input is the last user message, calls are flattened across turns
// Deterministic: input is the last user message, calls are flattened across turns
316
+
val deterministic = trajectory.toTestCase(tools)
317
+
val validity =ToolCallValidityEvaluator.builder().build().evaluate(deterministic)
318
+
319
+
// Judge: input is the transcript (tool calls name-only), tasks listed in metadata
320
+
val judgeCase = trajectory.toTestCase(tools, listOf("Check weather", "Book a hotel"))
321
+
val completion =TaskCompletionEvaluator.builder().judge(judgeLM).build().evaluate(judgeCase)
322
+
```
323
+
324
+
</TabItem>
325
+
</Tabs>
326
+
327
+
#### Tool Calls in the Transcript
328
+
329
+
`toText()` and `toJson()` render each turn's tool calls. `toText()` adds one compact `[tool: name(args)]` line per call under the message; `toJson()` adds a `toolCalls` array to a turn that has any. A tool-free conversation renders exactly as before, byte-identical, so adding tool calls to one turn never reshapes the rest.
330
+
331
+
To let the trajectory judge reason over tool usage, turn it on with `includeToolCalls(true)`. It is off by default, so existing judge suites see an unchanged prompt.
lead: "Tool calls on conversation turns: an assistant message now carries the tool calls it made, so a multi-turn conversation feeds the agent tool evaluators per turn, with no LLM. The trajectory judge can also reason over tool usage.",
13
+
groups: [
14
+
{
15
+
label: "Added",
16
+
items: [
17
+
{
18
+
title: "Tool calls on assistant turns",
19
+
body: (
20
+
<>
21
+
<code>Message</code> now carries a typed <code>List<ToolCall></code>, set via{" "}
22
+
<code>Message.assistant(content, toolCalls)</code> or the builder's{" "}
23
+
<code>assistantMessage(content, toolCalls)</code> (and the Kotlin DSL's{" "}
24
+
<code>assistant(content, toolCalls)</code>). A tool-free turn is unchanged, and the
25
+
three-argument <code>Message</code> constructor still resolves, so existing code
26
+
compiles and runs as before.
27
+
</>
28
+
),
29
+
},
30
+
{
31
+
title: "Per-turn tool evaluation",
32
+
body: (
33
+
<>
34
+
<code>ConversationTrajectory.toolCallsByTurn()</code> returns one tool-call list per
35
+
assistant turn (in order) to score each turn with the deterministic agent
36
+
evaluators, and <code>toolCalls()</code> flattens them into one list.{" "}
37
+
<code>toTestCase()</code>/<code>toTestCase(tools)</code> build a deterministic case
38
+
(input is the last user message), while <code>toTestCase(tools, tasks)</code> builds
39
+
the judge case for <code>TaskCompletionEvaluator</code> and{" "}
40
+
<code>ToolArgumentHallucinationEvaluator</code> over the whole conversation, with tool
41
+
calls rendered name-only so their arguments stay out of the hallucination grounding.
42
+
Also{" "}
43
+
<code>toAgentTrace()</code>/<code>toAgentOutputs()</code> for the standard agent
44
+
output map.
45
+
</>
46
+
),
47
+
},
48
+
{
49
+
title: "Tool calls in the trajectory transcript",
50
+
body: (
51
+
<>
52
+
<code>toText()</code> and <code>toJson()</code> render each turn's tool calls, and{" "}
53
+
<code>TrajectoryEvaluator.includeToolCalls(true)</code> adds them to the judge
54
+
prompt. Both are off by default for tool-free conversations, whose rendered output
0 commit comments