Runner: Improve robustness in solver results parsing

[danielelerede-oet]

Summary

This PR improves the robustness and correctness of solver result handling in run_solver.py, ensuring that invalid or incomplete solver runs are not misclassified as successful and that result parsing is safe across solver backends.

Motivation

During benchmarking, several issues were identified in how solver results were interpreted:

• Runs with invalid or unclear termination conditions (e.g. "unknown") could still be reported as successful
• Missing solver models (solver_model is None) could lead to crashes or invalid metric extraction
• MILP-specific metrics were computed even when not applicable
• Helper functions relied on assumptions that do not hold across all solver backends
• Some warning runs contained valid results and should not be discarded

These issues could lead to misleading benchmark results and inconsistent behaviour across solvers.

Changes

1. Improve classification of solver runs

• Runs with termination conditions such as "unknown", "error", "failed" or "aborted" are now classified as ER
• Objective values are cleared when no valid solution is available
• This prevents invalid runs from being treated as successful

2. Preserve valid warning runs

• Runs with "status": "warning" are no longer discarded automatically
• If a valid objective value is present, the result is kept
• Only warnings without a usable solution are promoted to ER

3. Handle missing solver models

• Added guards for cases where solver_model is None
• Applied consistently across:
  • runtime extraction
  • duality gap computation
  • MIP detection
  • MILP metric extraction
• Prevents crashes and avoids producing invalid metrics

4. Fix MILP metrics extraction

• get_milp_metrics now explicitly receives solver_name
• Metrics are only computed when:
  • HiGHS is available
  • the problem is actually a MIP
  • a valid solver model exists

5. Restrict MILP metric computation to MIP problems

• Avoid computing duality gap and integrality violation for non-MIP problems
• Ensures correctness of reported metrics

6. Improve robustness of helper functions

• get_duality_gap and is_mip_problem now safely handle missing solver models
• Avoid reliance on implicit assumptions about solver outputs

7. Fix issues in HiGHS-HiPO runner

• Removed reliance on implicit global variables (solver_name)
• Improved function signature for clarity and correctness

Impact

• Prevents invalid solver runs from contaminating benchmark results
• Preserves usable results that were previously discarded due to "warning" status
• Improves consistency and reliability of result parsing across solvers
• Makes the code more robust to solver-specific behaviour and incomplete outputs

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
solver-benchmark	Ready	Preview, Comment	Jul 15, 2026 8:27am

[eantonini]

Everything looks good from my side. Thanks for improving the robustness in result parsing.

[siddharth-krishna]

Sorry for the extremely slow review! This looks great and will improve our benchmarking robustness, thank you. Now that all v2 runs are complete we can merge this in. :)

[danielelerede-oet]

@eantonini the CI failure appears to be indeed related to the changes introduced in this PR. Before these changes, some cases were apparently not classified as errors, whereas they are now reported with Status=ER.

In particular, the sample benchmark job fails because FINE-hydrogen-partload with SCIP now ends with Status=ER
and Termination Condition=unknown while the CI currently expects every non-GLPK run to have Status=ok.

One idea could be to update the CI logic to account for these cases too. cc @siddharth-krishna

[eantonini]

@danielelerede-oet If it is only one sample problem that fails, you can remove it from the CI.

[danielelerede-oet]

@danielelerede-oet If it is only one sample problem that fails, you can remove it from the CI.

Not really. There are two issues affecting GLPK and SCIP:

• SCIP: the FINE-hydrogen-partload benchmark completes successfully and SCIP reports a valid solution (Solution parsed successfully), but Linopy returns termination_condition = "unknown". The benchmark runner currently maps any "unknown" termination condition to Status = ER, causing the benchmark to be reported as failed despite a valid solution apparently being available.
• GLPK: several benchmarks (all SWITCH models and temoa-utopia) fail while parsing the solver output. I'm quite sure that the exception originates in Linopy when converting the objective value:
  ValueError: could not convert string to float: 'ee139541670.1' (fix: GLPK objective parsing PyPSA/linopy#818)

[danielelerede-oet]

@eantonini the problem with SCIP is only related to the linopy version in our test environment. The newest release (0.8.0) is already parsing SCIP outputs correctly. Given that my PR to fix GLPK objective parsing has been merged, I would wait for the newest release to update the env (cc @siddharth-krishna )