tool-evaluation-plugins

Tool-Call F1 Scoring

script grader plugins that compute F1 scores over tool calls, comparing expected tools against actual agent behavior.

Graders

graders/tool-call-f1.ts — Name-only F1

Computes precision, recall, and F1 by comparing expected tool names against actual tool calls from output.

• True positive: expected tool was called
• False negative: expected tool was NOT called
• False positive: unexpected tool was called

graders:
  - name: tool-f1
    type: script
    command: ["bun", "run", "../graders/tool-call-f1.ts"]
    expected_tools: ["search", "fetch"]

graders/tool-args-f1.ts — Name + argument F1

Extends the name-only grader by also validating tool arguments. A call is a hit only if both the name matches AND the required arguments are present (subset match).

graders:
  - name: tool-args-f1
    type: script
    command: ["bun", "run", "../graders/tool-args-f1.ts"]
    expected_tools:
      - tool: search
        args: { query: "weather tokyo" }
      - tool: fetch

Running

cd examples/features/tool-evaluation-plugins
bun agentv eval evals/suite.yaml --target <your-target>

Output

Each grader returns:

{
  "score": 0.667,
  "assertions": [
    { "text": "Expected tool 'search' was called", "passed": true },
    { "text": "Expected tool 'fetch' was NOT called", "passed": false }
  ],
  "details": { "precision": 1, "recall": 0.5, "f1": 0.667, "tp": 1, "fp": 0, "fn": 1 }
}

When to Use

Need	Solution
Exact tool sequence	Built-in tool_trajectory with mode: in_order
Minimum tool counts	Built-in tool_trajectory with minimums
Set-based F1 scoring	This plugin (tool-call-f1.ts)
F1 with argument validation	This plugin (tool-args-f1.ts)