You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Update metadata and summary in paper.md
* Update paper.bib
* Update metadata and summary in paper.md
* Update paper/paper.md
* Update paper/paper.md
* Update paper metadata and summary content
Copy file name to clipboardExpand all lines: paper/paper.md
+15-8Lines changed: 15 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ authors:
11
11
orcid: 0000-0001-7729-2513
12
12
affiliation: 1
13
13
- name: Dominique Monnet
14
-
orcid:
14
+
orcid:0000-0002-5482-6831
15
15
affiliation: 3
16
16
- name: Dominique Orban
17
17
orcid: 0000-0002-8017-7687
@@ -40,20 +40,20 @@ bibliography: paper.bib
40
40
where $f:\mathbb{R}^n \rightarrow \mathbb{R}$ is a continuously differentiable function, with $\ell \in \left(\mathbb{R} \cup \{-\infty\} \right)^n$, and $u \in \left(\mathbb{R} \cup \{+\infty\} \right)^n$.
41
41
The algorithms implemented here are iterative methods that aim to compute a stationary point of \eqref{eq:nlp} using first and, if possible, second-order derivatives.
42
42
43
-
Our initial motivation for considering \eqref{eq:nlp} and developing `JSOSolvers.jl` is to solve large-scale unconstrained and bound-constrained problems such as parameter estimation in inverse problems, design optimization in engineering, and regularized machine learning models, and use these solvers to solve subproblems of penalty algorithms, such as `Percival.jl`[@Percival_jl] or `FletcherPenaltySolver.jl`[@FletcherPenaltySolver_jl], for constrained nonlinear continuous optimization problems.
43
+
Our initial motivation for considering \eqref{eq:nlp} and developing `JSOSolvers.jl` is to solve large-scale unconstrained and bound-constrained problems such as parameter estimation in inverse problems, design optimization in engineering, and regularized machine learning models, and use these solvers to solve subproblems of penalty algorithms, such as `Percival.jl`[@Percival_jl] or `FletcherPenaltySolver.jl`[@FletcherPenaltySolver_jl], for constrained nonlinear continuous optimization problems.
44
44
In many of these problems, explicitly storing Hessian matrices is either computationally prohibitive or impractical.
45
45
The solvers in `JSOSolvers.jl` adopt a matrix-free approach, where standard optimization methods are implemented without forming derivative matrices explicitly.
46
46
This strategy enables the solution of large-scale problems even when function and gradient evaluations are expensive.
47
47
48
48
The library includes TRON, a trust-region Newton method for bound-constrained problems following the classical formulation of @tron, TRUNK, a factorization-free trust-region Newton method based on the truncated conjugate gradient method, as described by @conn2000trust, an implementation of L-BFGS, a limited-memory quasi-Newton method using a line search globalization strategy, and FOMO, a first-order method based on quadratic regularization designed for unconstrained optimization.
49
-
The latter is an extension of a quadratic regularization method described by @aravkin2022proximal, and called R2 in `JSOSolvers.jl`.
49
+
FOMO is an extension of a quadratic regularization method described by @aravkin2022proximal, and called R2 in `JSOSolvers.jl`.
50
50
Unlike textbook implementations, our solvers introduce several design differences.
51
51
TRON operates in a factorization-free mode, while the original Fortran TRON requires an explicitly formed Hessian.
52
52
TRUNK departs from the Conn–Gould–Toint formulation by supporting a non-monotone mode and multiple subproblem solvers (CG, CR, MINRES, etc.).
53
53
Our L-BFGS implementation uses a simplified line-search strategy that avoids the standard Wolfe conditions while maintaining robust convergence in practice.
54
54
55
55
A nonlinear least-squares problem is a special case of \eqref{eq:nlp}, where $f(x)=\frac{1}{2}\|F(x)\|^2_2$ and the residual $F:\mathbb{R}^n \rightarrow \mathbb{R}^m$ is continuously differentiable, which appears in many applications, including inverse problems in imaging, geophysics, and machine learning. While it is possible to solve the problem using only the objective, knowing $F$ independently allows the development of more efficient methods.
56
-
TRON and TRUNK have specialized implementations leveraging the structure of residual models to improve performance and scalability.
56
+
Specialized variants of TRON and TRUNK, called TRON-NLS and TRUNK-NLS, leverage the structure of residual models to improve performance and scalability.
57
57
58
58
A key strength of `JSOSolvers.jl` lies in its efficiency and flexibility.
59
59
The solvers support fully in-place execution, allowing repeated solves without additional memory allocation, which is particularly beneficial in high-performance and GPU computing environments where memory management is critical.
@@ -62,13 +62,15 @@ Moreover, TRUNK, TRUNK-NLS, and FOMO support GPU arrays, broadening the range of
62
62
The package documentation and \url{https://jso.dev/tutorials} provide examples illustrating the use of different floating-point systems.
63
63
Furthermore, the solvers expose in-place function variants, allowing multiple optimization problems with identical dimensions and data types to be solved efficiently without reallocations.
64
64
65
-
`JSOSolvers.jl` is built upon the JuliaSmoothOptimizers (JSO) tools [@The_JuliaSmoothOptimizers_Ecosystem].
65
+
`JSOSolvers.jl` is built upon the JuliaSmoothOptimizers (JSO) tools [^jso].
66
66
JSO is an academic organization containing a collection of Julia packages for nonlinear optimization software development, testing, and benchmarking.
67
67
It provides tools for building models, accessing problem repositories, and solving subproblems.
68
68
Solvers in `JSOSolvers.jl` take as input an `AbstractNLPModel`, JSO's general model API defined in `NLPModels.jl`[@NLPModels_jl], a flexible data type to evaluate objective and constraints, their derivatives, and to provide any information that a solver might request from a model.
69
69
The user can hand-code derivatives, use automatic differentiation, or use JSO-interfaces to classical mathematical optimization modeling languages such as AMPL [@fourer2003ampl], CUTEst [@cutest], or JuMP [@jump].
70
70
The solvers rely heavily on iterative linear algebra methods from `Krylov.jl`[@Krylov_jl].
71
71
72
+
[^jso]: JuliaSmoothOptimizers https://jso.dev/
73
+
72
74
# Statement of need
73
75
74
76
Julia's JIT compiler is attractive for the design of efficient scientific computing software, and, in particular, mathematical optimization [@lubin2015computing], and has become a natural choice for developing new solvers.
@@ -84,19 +86,23 @@ Several alternatives to `JSOSolvers.jl` are available within and outside the Jul
84
86
`Optim.jl`[@mogensen2018optim] is a general-purpose optimization library in pure Julia, suitable for small to medium-scale problems, but it lacks in-place execution and GPU support.
85
87
`NLopt.jl`[@NLopt] provides access to a broad collection of optimization algorithms via a C library but does not support matrix-free methods or extended precision.
86
88
`AdaptiveRegularization.jl`[@AdaptiveRegularization_jl] offers a matrix-free, multi-precision solver for unconstrained problems and is closely aligned with the design philosophy of `JSOSolvers.jl`.
87
-
Ipopt [@wachter2006implementation], via `Ipopt.jl`, is widely used and efficient but requires explicit derivatives and is limited to CPU execution.
89
+
Ipopt [@wachter2006implementation], via `Ipopt.jl`, is a widely used and efficient solver, but requires explicit derivatives and is limited to CPU execution.
88
90
GALAHAD [@galahad], a Fortran-based suite for large-scale problems, is accessible through experimental Julia wrappers, yet lacks native composability.
89
91
Commercial solvers such as Artelys Knitro [@byrd2006k] provide robust algorithms but remain constrained by licensing and limited Julia interoperability.
90
92
`Optimization.jl` is a wrapper to existing optimization packages.
91
93
92
94
## Benchmarking
93
95
94
96
`JSOSolvers.jl` can solve large-scale problems and can be benchmarked easily against other JSO-compliant solvers using `SolverBenchmark.jl`[@SolverBenchmark_jl].
95
-
We include below performance profiles [@dolan2002benchmarking] with respect to elapsed time of `JSOSolvers.jl` solvers against Ipopt on all the 291 unconstrained problems from the CUTEst collection [@cutest], whose dimensions range from 2 up to 192,627 variables.
96
-
LBFGS uses only first-order information, while TRON and TRUNK use Hessian-vector products and IPOPT uses the Hessian as a matrix.
97
+
We include below performance profiles [@dolan2002benchmarking] with respect to elapsed time of `JSOSolvers.jl` solvers against Ipopt on all the 291 unconstrained problems from the CUTEst collection [@cutest], whose dimensions range from 2 up to 192,627 variables.[^hardware]
98
+
LBFGS uses only first-order information, while TRON and TRUNK use Hessian-vector products and Ipopt uses the Hessian as a matrix.
97
99
Without explaining performance profiles in full detail, the plot shows that Ipopt is fastest on 42 problems (15%), TRON on 9 (3%), TRUNK on 64 (21%), and L-BFGS on 176 (60%).
98
100
Nearly all problems were solved within the 20-minute limit: TRON solved 272 (93%), Ipopt 270, TRUNK 269, and L-BFGS 267.
99
101
102
+
[^hardware]: Benchmarks were run sequentially on a CPU-only machine.
103
+
The hardware configuration was an Intel Core i7-class processor with approximately
104
+
16 GB of RAM running Linux. Timings are intended for relative comparison only.
105
+
100
106
Overall, these results are encouraging.
101
107
Although Ipopt is a mature and highly optimized solver, TRUNK and L-BFGS achieve comparable problem coverage while being significantly faster on many instances.
102
108
This suggests that the algorithms implemented here are competitive for large-scale problems.
@@ -118,3 +124,4 @@ Dominique Orban is partially supported by an NSERC Discovery Grant.
0 commit comments