benchmarks: convergence suite v1 (Ipopt + MadNLP on X gate)#93
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced May 20, 2026
jack-champagne
added a commit
that referenced
this pull request
May 20, 2026
After the deterministic+sweep rework no testitems use :experimental anymore, and the sweeps add ~5 seconds in aggregate on top of a ~10-minute matrix — not worth a parallel filter framework. Drops both INCLUDE_EXPERIMENTAL and INCLUDE_ROBUSTNESS env vars and the ti.tags closure. The robustness sweeps now run on every PR (along with their deterministic baselines). Keeps a `/benchmark/` path filter following Piccolissimo.jl's convention — the benchmark subtree has its own Project.toml + deps + workflow, so its @testItems shouldn't be discovered by `@run_package_tests`. This also preemptively unbreaks PR #93's CI (which fails today because benchmark/convergence/convergence.jl is being picked up). Verified locally on Julia 1.12: 355 pass / 1 broken / 0 fail. TimeConsistency sweep: pass_rate = 1.0 (20/20) NonlinearKnotPoint sweep: pass_rate = 0.8 (16/20, threshold 0.65) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jack-champagne
added a commit
that referenced
this pull request
Jun 8, 2026
Re-cut of #93 onto current main. The branch had gone stale (it predated #75 + src changes), so a direct merge would have deleted #75's timing suite/docs and reverted src/ — this re-cut adds ONLY the convergence suite + its workflow. - benchmark/convergence/: Ipopt + MadNLP X-gate convergence testitems (InfidelityConvergence + ipopt_capture), report.jl emitting the /bench-convergence dashboard JSON + job-summary table (reuses the shared benchmark/BenchmarkUtils.jl from #75). README frames it as a sanity baseline. - .github/workflows/benchmark-convergence.yml. - Dropped the runtests.jl change: main already excludes benchmark/ testitems (the filter landed with #75). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jack-champagne
force-pushed
the
benchmarks/convergence-suite-v1
branch
from
b1a48b8 to
248dde9
Compare
… from timing run) Re-cut onto current main (post #75 + #101). Adds the Ipopt+MadNLP X-gate convergence suite (own benchmark-convergence.yml + /bench-convergence dashboard, reusing the shared benchmark/BenchmarkUtils.jl) and extends the timing benchmark.yml filter to skip BOTH alloc_profile and benchmark/convergence/ — those suites have their own workflows + project envs, so the timing run picking them up used the wrong env / blew the 60-min budget (the prior failure here). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jack-champagne
force-pushed
the
benchmarks/convergence-suite-v1
branch
from
248dde9 to
d106b7a
Compare
The docs job had no timeout, so it defaulted to 360 min. On 2026-06-08 a deploydocs gh-pages push stalled and ran 90+ min, and because docs.yml's concurrency group does not cancel in-progress main builds, it queued the next main docs deploy behind it. makedocs is ~5 min; 30 min bounds build+deploy with margin and fails fast on a deploy stall instead of blocking the queue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
benchmark/convergence/— a small DTO benchmark environment thatexercises HarmoniqsBenchmarks.jl v0.2.0's new convergence API
(
InfidelityConvergence,ipopt_capture,compare_convergence).Scope is deliberately tight (one IPOPT + one MadNLP testitem on the same
X-gate state-transfer); atoms-demo / spin-qubit-demo / bosonic-demo problems
land in separate follow-up PRs once this harness is proven green.
Depends on: HarmoniqsBenchmarks.jl
main(v0.2.0 + convergence schema, not yet registered).What's included
benchmark/convergence/Project.toml— isolated env with HBJ (main),Ipopt, MadNLP, NamedTrajectories.
DirectTrajOptpulled in as a path depto
../..so we test the package under PR HEAD, not General.benchmark/convergence/problem_utils.jl—_make_xgate_prob(bilinear4D rep + terminal cost + control regularizers) and
_build_convergence_result(rebuilds aBenchmarkResultwith aConvergenceCriterionattached).benchmark/convergence/convergence.jl— two@testitems:ipopt_capture()extracts finaliter_count+inf_pr, buildsInfidelityConvergencewithtarget_infidelity = 1e-3,feas_tol = 1e-6, assertsconverged(...).post-solve
evaluator.constraint_violationas the primal-infeasibilityproxy (no MadNLP capture analogue yet).
benchmark/convergence/README.md— usage instructions..github/workflows/benchmark-convergence.yml— Julia 1.12, 30 mintimeout, JLD2 artifact upload.
Local results
JLD2 artifacts produced:
xgate_convergence_ipopt_N51_<commit>.jld2xgate_convergence_madnlp_N51_<commit>.jld2Test plan
benchmark-convergenceworkflow green on this PRconverged(result.convergence) == truecompare_convergence([result])returns one row per converging benchmarkOut of scope (follow-up PRs)
ipopt_capture()