[FEAT]: Benchmark Module Using pytest-benchmark to Compare Pipeline Performance

[marcvergees]

📝 Description

Build a comprehensive Python benchmark module using pytest-benchmark to evaluate and empirically compare Pipeline A (Canonical Incident Schema) vs Pipeline B (Template-Driven Extraction). This will measure extraction accuracy, consistency, hallucination rates, latency, cost, and scalability across both pipelines.

💡 Rationale

Following discussion #611, we agreed that the best way to settle the debate between Pipeline A and Pipeline B is through empirical evidence rather than assumptions. To make this decision objectively, we need robust benchmarking infrastructure that measures:

• Extraction Accuracy: Field-level precision against ground truth
• Consistency: Cross-form consistency (Pipeline A) and cross-template field consistency (Pipeline B)
• Hallucination Rate: Detection of fabricated values in critical fields (injury type, cause, property damage)
• Performance: End-to-end latency, LLM call counts, and token costs
• Scalability: Performance degradation when generating 1, 10, and 100 forms from the same narrative

🛠️ Proposed Solution

Implement a structured benchmark framework with pytest-benchmark integration:

benchmark/
├── README.md
├── conftest.py
├── run.py
│
├── datasets/
│   ├── narratives/          # Synthetic + anonymized incident reports
│   ├── templates/           # Real-world forms (NFIRS, MA, CAL FIRE, NY)
│   └── ground_truth/        # Expected extraction results
│
├── pipelines/
│   ├── base.py              # Abstract pipeline interface
│   ├── pipeline_a.py        # Canonical Incident Schema wrapper
│   └── pipeline_b.py        # Template-Driven Extraction wrapper
│
├── evaluators/
│   ├── accuracy.py          # Field-level accuracy (LLM-as-judge)
│   ├── consistency.py       # Cross-form/cross-template consistency
│   ├── hallucination.py     # False positive field detection
│   └── aggregate.py         # Combined metrics reporting
│
├── tests/
│   ├── test_pipeline_a_benchmark.py
│   ├── test_pipeline_b_benchmark.py
│   └── test_extraction_metrics.py
│
├── reports/
│   ├── raw/                 # Benchmark run outputs
│   ├── summaries/           # CSV/JSON comparison reports
│   └── figures/             # Visualization artifacts
│
├── config/
│   └── benchmark.yaml       # Configuration for metrics, dataset size
│
└── utils/
    ├── io.py                # File/data I/O helpers
    ├── metrics.py           # Shared metric calculations
    └── logging.py           # Structured benchmark logging

Implementation approach:

• Design ground-truth schema (flat field mapping + structured by NERIS module with explicit nulls)
• Create initial dataset: 10 narratives × 4 templates = 40 benchmark cases (synthetic + anonymized real reports)
• Implement pipeline wrappers using existing services
• Build pytest-benchmark test suite with custom fixtures
• Develop evaluator modules (accuracy with LLM-as-judge, consistency checkers, hallucination detection)
• CLI runner (python run.py --pipeline a --output reports/) + GitHub Actions integration
• Generate comparison reports (latency, cost, scaling analysis)

✅ Acceptance Criteria

• Benchmark module executes via pytest and generates performance reports
• Both pipelines can be run independently with configurable dataset
• Extraction accuracy measured against ground truth with normalization (RapidFuzz for partial matches)
• Consistency metrics computed for shared fields across outputs
• Hallucination detection distinguishes null returns from false positives
• Latency and token cost tracked per incident and aggregated
• Scalability tests run with 1/10/100 form generation
• Reports generated in CSV/JSON format with markdown summaries
• CLI usage: python benchmark/run.py --pipeline a --dataset 10 --output reports/
• CI integration: benchmark runs on PR branches without blocking merge
• Documentation in benchmark/README.md with dataset schema and metric definitions

📌 Additional Context

• Related discussion: Designing a Benchmark Module to Evaluate Pipeline Improvements #611 "Designing a Benchmark Module to Evaluate Pipeline Improvements"
• Inspired by collaborative feedback from @chetanr25, @abhishek-8081, @vharkins1
• Ground truth should support confidence scores per field for accuracy vs confidence analysis
• Consider LLM-as-a-judge for nuanced partial value matching (e.g., "Floor 3" ≈ "3rd floor")
• Recommend CLI for development testing + CI protection of main branch
• Initial benchmark dataset should combine fully synthetic narratives with anonymized real-world fire reports