solver benchmark: specifying solver-specific options (#152)

* solver benchmark: specifying solver-specific options

* Redoing index.jmd and Project.toml to fix SolverBenchmark tutorial

* Apply suggestions from code review

Co-authored-by: Tangi Migot <tangi.migot@gmail.com>

* changes acc. to review

* Fix newline issue at end of Project.toml

* Update tutorials/introduction-to-solverbenchmark/index.jmd

* Update tutorials/introduction-to-solverbenchmark/index.jmd

* Apply suggestions from code review

Co-authored-by: Tangi Migot <tangi.migot@gmail.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tutorials/introduction-to-solverbenchmark/index.jmd

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* NLPModelsTest

* Update index.jmd

* Fix formatting for accessing isConvex column

---------

Co-authored-by: Tangi Migot <tangi.migot@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: 3 people

tutorials/introduction-to-solverbenchmark/Project.toml

@@ -1,14 +1,18 @@
 [deps]
 DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
+NLPModelsTest = "7998695d-6960-4d3a-85c4-e1bceb8cd856"
 Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
 Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
 PyPlot = "d330b81b-6aea-500a-939a-2ce795aea3ee"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 SolverBenchmark = "581a75fa-a23a-52d0-a590-d6201de2218a"
+SolverCore = "ff4d7338-4cf1-434d-91df-b86cb86fb843"
 
 [compat]
 
 DataFrames = "1.3.4"
+NLPModelsTest = "0.9"
 Plots = "1.31.7"
 PyPlot = "2.10.0"
 SolverBenchmark = "0.5.3"
+SolverCore = "0.3"

tutorials/introduction-to-solverbenchmark/index.jmd

@@ -221,3 +221,26 @@ p = profile_solvers(stats, costs, costnames)
 Here is a useful tutorial on how to use the benchmark with specific solver:
 [Run a benchmark with OptimizationProblems](https://jso.dev/OptimizationProblems.jl/dev/benchmark/)
 The tutorial covers how to use the problems from `OptimizationProblems` to run a benchmark for unconstrained optimization.
+
+### Handling `solver_specific` in `stats`
+
+If a solver's `GenericExecutionStats` contains a `solver_specific` dictionary, then when `bmark_solvers` processes the results it creates a column in the per-solver `DataFrame` for each key in that dictionary. These columns can then be analyzed and compared alongside the standard metrics such as `status` and `elapsed_time`.
+
+Here is an example showing how to set a solver-specific flag so that it appears as a column in the resulting stats table and can be used for tabulation:
+```julia
+using NLPModelsTest, DataFrames, SolverCore, SolverBenchmark
+
+function newton(nlp)
+  stats = GenericExecutionStats(nlp)
+  set_solver_specific!(stats, :isConvex, true)
+  return stats
+end
+
+solvers = Dict(:newton => newton)
+problems = [NLPModelsTest.BROWNDEN()]
+stats = bmark_solvers(solvers, problems)
+
+# Access the solver-specific column `:isConvex` for the `:newton` solver
+df_newton = stats[:newton]
+df_newton.isConvex
+```