Define the benchmarks

[imalsogreg]

What does one benchmark (a test run across each model) look like?

A proposal:

Benchmarks have:

• A name
• One BAML function template. The template is parameterized over some independent variables
• Several tags

What are the independent variables? They're variables whose impact on the benchmark we wish to measure:

• Which model (this is the most important independent variable)
• Whether or not we ask for CoT via the prompt
• Whether or not we use aggressive tricks to reduce token count, like aliasing field names to shorter forms and eliding object key quotation marks.

What's a tag? A free-form single-word label that can be used for selecting subsets of benchmarks.

tag = "Agent" | "Multi-Tool" | "Productivity" | "Computer Use" | "Code Assistant" | ...

[George-RD]

On the metrics page, I suggested we determine a $ / 1000 successful parses value. This could be interesting to also calculate based on the aggressive tricks for reducing token count. - To provide analytics for this, would make sense to have a series of the same test with same expected result

Agree COT. I was thinking of having it as a separate entry as it would be useful on a table to see the COT enabled/disabled versions of the same model in their actual rank order.

But could also be interesting if it is a property to show case comparisons of models with and without their COT enabled.

How should we handle prompt based COT vs models with innate/toggle able COT?

Tags 100%.

Also Tags for the suggested test difficulty that would be calculated in metrics