feat: add self-improving benchmark pipeline for OAPE tools

[rausingh-rh]

Summary

Adds a self-improving benchmark pipeline that measures and iteratively improves the quality of the OAPE code generation tools (api-generate, api-implement) by comparing their output against real, already-merged EP implementations.

How it works

1. Feedback loop: Generate code -> compare vs ground truth -> improve tool instructions -> repeat (default 3 iterations)
2. Bias-free: Repo cloned at pre-EP commit so the agent never sees the actual implementation
3. Generic improvements: Tool instruction edits are pattern-based, not EP-specific, so improvements benefit all future EPs
4. Multi-PR support: Combines related PRs into single ground truth for EPs implemented across multiple PRs

Metrics

The report uses actionable metrics instead of abstract scores:

Metric	What it measures
Completeness	What % of the human's structs, fields, and functions did the tool also generate?
Convention	What % of kubebuilder markers match between generated and ground truth?
Build	Did make build pass on the generated code?
Matched	Files the tool generated that also exist in the human implementation (true positives)
Missed	Files in the human implementation that the tool did NOT generate (gaps to close)
Wrong	Files the tool touched that it should not have -- unrelated to the EP (real errors)
Extras	Useful files the tool generated that the human didn't -- tests, samples, validation (tool outperformed human)
Auto	Files auto-generated by make generate/make manifests (expected build artifacts)

Example report table:

| Iter | Tool Version | Completeness | Convention | Build | Matched | Missed | Wrong | Extras | Auto |
|------|-------------|-------------|------------|-------|---------|--------|-------|--------|------|
| 1    | original    | 91.0%       | 97.8%      | PASS  | 6       | 12     | 1     | 5      | 0    |
| 2    | improved-v1 | 99.5%       | 97.8%      | PASS  | 8       | 10     | 1     | 5      | 0    |
| 3    | improved-v2 | 94.2%       | 97.8%      | PASS  | 9       | 9      | 1     | 6      | 0    |

How others can use it

cd benchmark/

# 1. Edit config.yaml with your EP-to-implementation mappings
# 2. Run the feedback loop
python benchmark.py run --config config.yaml

# 3. Review results in benchmark/results/<repo>/ep-<number>/report.md
# 4. Optionally push a good iteration as a PR
python benchmark.py push --ep 1863 --repo <fork-url> --iteration 3 --base-branch main

See benchmark/README.md for full documentation.

Tested on

EP	Repo	Best Completeness	Convention	Build	Wrong Files
#1863	zero-trust-workload-identity-manager	97.2%	99.6%	PASS	1
#1834	external-secrets-operator	99.5%	97.8%	PASS	1

New files

File	Purpose
benchmark/benchmark.py	CLI entry point with run, report, full, push subcommands
benchmark/runner.py	Generation + improvement feedback loop via Claude Agent SDK
benchmark/compare.py	Delta-aware diff engine, scoring, file classification
benchmark/isolate.py	PR timeline resolution, bias-free environment setup
benchmark/ground_truth.py	Combined truth extraction from multiple PRs
benchmark/report.py	Markdown + JSON report generation
benchmark/models.py	Shared data models
benchmark/go_ast_helper/	Go AST extraction for struct-level comparison
benchmark/config.yaml	Example config
benchmark/README.md	Full usage documentation with metric definitions

Test plan

• Run benchmark on EP #1863 (ZTWIM) with PRs [68, 82] -- 3 iterations, all PASS
• Run benchmark on EP #1834 (ESO) with PRs [67, 74] -- 3 iterations, all PASS
• Verify bias prevention (no EP code at baseline commit)
• Verify tool improvements are generic (not EP-specific)
• Verify original tool files restored after benchmark
• Verify make build passes on generated code
• PR raised from generated output: feat: add SPIRE federation support (EP #1863) - OAPE benchmark improved-v2 rausingh-rh/zero-trust-workload-identity-manager#8

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rausingh-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [rausingh-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment