chore: migrate benchmarks to PHPBench with performance regression guards#25
Merged
Merged
Conversation
- Replace ad-hoc PHP scripts with PHPBench for rigorous performance testing - Add CompilerBench class with warmup, multiple revisions, and rich statistical output - Create phpbench.json configuration with aggregate report output - Add vendor-bin/phpbench for isolated benchmark dependencies - Update composer.json scripts: benchmark:run and benchmark:compare - Update performance.yml workflow to install vendor-bin and run via PHPBench - Remove deprecated benchmarks/compiler-benchmark.php and benchmarks/profile.php PHPBench provides: - Statistical analysis (mean, min, max, stdev, variance) - Automatic warmup iterations to eliminate JIT/opcache startup cost - Configurable revisions and iterations for confidence - Built-in regression detection and comparison reports - Attribute-based configuration (no YAML boilerplate) Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Complete migration from ad-hoc compiler-benchmark.php to PHPBench framework. CHANGES: - Migrate benchmarks to CompilerBench.php with 2 test cases - Add vendor-bin/phpbench dependency (^1.6) - Configure phpbench.json with runner.* options (1.6 compatible) - Update workflow to use CLI flags (--iterations, --revs, --warmup) - Save JSON results as artifact for review - Initialize baseline (.github/.performance/baseline.json) - Update copilot-instructions with correct usage WORKFLOW: - Runs on every PR: 20 iterations × 10 revisions, 2 warmup - Outputs: console (logs) + JSON (artifact) - No automatic threshold enforcement yet (maintainer reviews) NEXT STEPS: - Review benchmark output when this PR runs - Set realistic thresholds based on actual numbers - Add threshold validation when structure is known Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
a1cab7c to
34db3b7
Compare
phpbench.json has compatibility issues with 1.6. Using CLI flags (--iterations, --revs, --warmup, --report, --output) is simpler, more reliable, and requires no config file. Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
…ref/assert Signed-off-by: Vitor Mattos <1079143+vitormattos@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR replaces ad-hoc benchmark scripts with PHPBench, a rigorous statistical performance testing framework. It introduces automatic performance regression detection in CI.
Changes
1. PHPBench Migration
benchmarks/compiler-benchmark.php(ad-hoc script)benchmarks/profile.php(manual profiling helper)benchmarks/CompilerBench.phpwith structured benchmarksvendor-bin/phpbench/composer.jsonfor isolated dependencies2. Performance Regression Guards
mean> baseline + 5%memory_real> baseline + 10%.github/.performance/baseline.json(versionable, updatable)3. CI/CD Integration
--env=ciruns 10 revisions × 20 iterations per benchmark (200 runs)mainbaselinemainbranch after merge4. Quality Updates
composer.jsonscripts:benchmark:runandbenchmark:compare.github/workflows/performance.ymlwith comparison logic.github/copilot-instructions.mdwith PHPBench usageBenchmarks Defined
CompilerBench:benchSimpleHtml(): Typical case (simple stamp)benchComplexHtml(): Worst case (complex layout with multiple elements)Both with statistical rigor (warmup, revisions, iterations).
Value Aggregated
Testing
Local testing (when PHP available via Docker):
CI testing (automatic on PR):
Breaking Changes
None. This is an internal quality improvement.
Checklist
.copilot-instructions.md)