Skip to content

README.md

Latest commit

 

History

History
267 lines (191 loc) · 5.83 KB

README.md

File metadata and controls

267 lines (191 loc) · 5.83 KB
Raw

Step 2: Tax Agent Benchmarking

Step 2 implements automated benchmarking for the tax agent using Runloop's benchmarking framework. This allows you to systematically evaluate the agent's performance across multiple test scenarios.

Overview

The step2 scripts create and run benchmarks that test the tax agent's ability to:

  • Parse different W-2 formats (PDF and text)
  • Extract key fields accurately
  • Calculate federal taxes correctly
  • Generate valid Form 1040 output

Prerequisites

  1. Complete Step 1 setup:

    pnpm step1_runloop_setup

  2. Set environment variables:

    export RUNLOOP_API_KEY=your-runloop-api-key
export OPENAI_API_KEY=your-openai-api-key

Quick Start

To set up and run benchmarks:

# Create scenarios and benchmark (one-time setup)
pnpm step2_setup

# Run the benchmark
pnpm step2_run_benchmark

Scripts

1. Create Scenarios (step2_create_scenarios)

Creates individual test scenarios for the tax agent:

pnpm step2_create_scenarios

What it does:

  • Creates 4 test scenarios covering different W-2 formats and income levels
  • Each scenario includes:
    • Problem statement (what the agent should do)
    • Expected values for validation
    • Scoring functions to evaluate success
    • Environment setup (devbox with agent)
  • Saves scenario IDs to frontend/resources/step2-scenarios.json

Scenarios created:

  1. basic-w2-parsing: Parse a simple text W-2 (Jane Doe, $72,500)
  2. high-income-calculation: High-income taxpayer (Mary Jones, $125,000)
  3. pdf-w2-parsing: Parse a PDF W-2 (E. Darling, $44,629)
  4. middle-income-calculation: Middle-income taxpayer (Billy Kidd, $95,000)

2. Create Benchmark (step2_create_benchmark)

Creates a benchmark from the scenarios:

pnpm step2_create_benchmark

What it does:

  • Loads the scenarios created in step 1
  • Creates a benchmark containing all scenarios
  • Saves benchmark ID to frontend/resources/step2-benchmark.json

3. Run Benchmark (step2_run_benchmark)

Runs the benchmark and monitors progress:

pnpm step2_run_benchmark

What it does:

  • Starts a benchmark run with all scenarios
  • Monitors progress (polls every 5 seconds)
  • Displays results for each scenario
  • Calculates overall statistics:
    • Total scenarios run
    • Number completed
    • Number passed (score > 70%)
    • Average score
  • Saves detailed results to frontend/resources/benchmark-run-{id}.json

Example output:

📊 Scenario Results:

  ✅ tax-agent-basic-w2-parsing
     Status: completed
     Score: 100.0%

  ✅ tax-agent-high-income-calculation
     Status: completed
     Score: 85.5%

  ❌ tax-agent-pdf-w2-parsing
     Status: completed
     Score: 65.0%

  ✅ tax-agent-middle-income-calculation
     Status: completed
     Score: 90.0%

📈 Overall Statistics:
  Total Scenarios: 4
  Completed: 4/4
  Passed (>70%): 3/4
  Average Score: 85.1%

Scoring Criteria

Each scenario is evaluated using three scoring functions:

  1. output-file-exists (30% weight)

    • Checks if the result.json file was created

  2. valid-json-output (30% weight)

    • Validates the output is well-formed JSON

  3. success-flag (40% weight)

    • Checks if the agent reported success in the output

A scenario passes if it achieves a score > 70%.

Configuration Files

frontend/resources/step2-scenarios.json

Contains scenario IDs and metadata:

{
  "scenarioIds": ["scn_123...", "scn_456..."],
  "scenarios": [
    {
      "name": "basic-w2-parsing",
      "description": "...",
      "id": "scn_123..."
    }
  ],
  "createdAt": "2024-12-16T..."
}

frontend/resources/step2-benchmark.json

Contains benchmark ID and configuration:

{
  "benchmarkId": "bmk_123...",
  "benchmarkName": "tax-agent-benchmark-...",
  "scenarioIds": ["scn_123...", "scn_456..."],
  "createdAt": "2024-12-16T..."
}

frontend/resources/benchmark-run-{id}.json

Contains detailed results from a benchmark run:

{
  "runId": "bmr_123...",
  "benchmarkId": "bmk_123...",
  "status": "completed",
  "scenarioRuns": [...],
  "statistics": {
    "total": 4,
    "completed": 4,
    "passed": 3,
    "averageScore": 0.851
  },
  "completedAt": "2024-12-16T..."
}

Customizing Scenarios

To add new test scenarios, edit packages/scripts/src/step2-create-scenarios.ts:

const TAX_SCENARIOS: TaxScenario[] = [
  // ... existing scenarios ...
  {
    name: 'your-new-scenario',
    description: 'Test description',
    w2File: 'your_w2.txt',
    expectedValues: {
      wages: 50000,
      federalWithholding: 5000,
    },
  },
];

Then re-run the setup:

pnpm step2_setup

Troubleshooting

"Scenarios configuration not found"

Run pnpm step2_create_scenarios first.

"Benchmark configuration not found"

Run pnpm step2_create_benchmark after creating scenarios.

"RUNLOOP_API_KEY environment variable is not set"

Make sure you've exported your Runloop API key:

export RUNLOOP_API_KEY=your-key-here

Scenarios failing with low scores

Check the detailed results in benchmark-run-{id}.json to see which scoring functions failed. Common issues:

  • Agent not creating output file
  • Invalid JSON format
  • Agent reporting failure in output

Next Steps

After running benchmarks:

  1. Review results in the saved JSON files
  2. Investigate failing scenarios
  3. Improve the tax agent based on findings
  4. Re-run benchmarks to measure improvements

API Reference

The step2 scripts use the Runloop SDK's benchmarking APIs:

  • runloop.scenarios.create() - Create test scenarios
  • runloop.benchmarks.create() - Create benchmarks
  • runloop.benchmarks.startRun() - Run benchmarks
  • runloop.benchmarks.runs.retrieve() - Check run status
  • runloop.benchmarks.runs.listScenarioRuns() - Get scenario results

Documentation: ~/src/api-client-ts/docs-md/@runloop/