Define the metrics

[imalsogreg]

What metrics do we collect for each (model * query)?

One quick proposal:

• Wall-clock time
• Parse succeeded
• Number of check failures (use checks as evals)
• Output tokens

[George-RD]

Excellent starting point! I'd add:

• Wall-clock time (perhaps normalized to tokens/s for easier comparison with other benchmarks)
• Models with reasoning/thinking modes should be separate entries
• Run n attempts per test per model, then report parse success as percentage(keep it at something like 10 for development, increase for actually benchmarking)
• Add columns for test type/difficulty categories (if I treat this metric thread to help define the output schema).

Would also like to track adjusted cost per successful parse:

• Success probability (p) as decimal
• Cost per attempt from input + average output tokens
• Adjusted cost per 1000 successful results = (1/p × cost per attempt) × 1000

This matters because when latency isn't critical, a less reliable but cheaper model might be preferable.

Finally, we should distinguish between parse success and parse accuracy. For tests with deterministic answers (like "extract the email from this text"), we can measure whether the successfully parsed output is actually correct. This gives us two metrics:

• Did it return valid structured data? (parse success)
• Was that data correct? (parse accuracy)

[George-RD]

Model	Tags	Success Rate	Avg Time (s)	Tokens/s	Output Tokens	Base Cost	Adjusted Cost/1k	Parse Accuracy
GPT-3.5	straightforward, extraction	90% (9/10)	1.2	125	150	$0.0003	$0.33	100%
GPT-3.5	medium, tool-calling, agent	70% (7/10)	2.1	95	200	$0.0004	$0.57	85%
Claude-Haiku	straightforward, extraction	85% (17/20)*	0.8	180	140	$0.0002	$0.24	95%
Llama-3-8B	complex, multi-tool, agent, code-gen	45% (9/20)	3.2	180	450	$0.0005	$1.11	78%

example table so far to discuss

[George-RD]

Thinking about it, rather than arbitrarily defining difficulty, we should calculate the test difficulty as property calculated based on overall benchmark data. These tags would need to be added as properties of the test in question, and metrics filterable as a subset of test results.

This will mean the benchmark would give us the ability to group various tests by difficulty:

Future potential extension: users input their BAML prompt in AI Studio/online. optional ability submit upload to the benchmark system, find similar BAML category and difficulty, and suggest potential models for the user to try. (don't want to re run the whole benchmark again every time one is submitted)