You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
where $f: \mathbb{R}^n \to \mathbb{R}$ and $c: \mathbb{R}^n \to \mathbb{R}^m$ are continuously differentiable, and $h: \mathbb{R}^n \to \mathbb{R} \cup \{+\infty\}$ is lower semi-continuous.
40
-
The nonsmooth objective $h$ can be a *regularizer* such as a sparsity-inducing penalty, model simple constraints such as $x$ belonging to a simple convex set, or be a combination of both.
40
+
The nonsmooth objective $h$ can be a *regularizer*, such as a sparsity-inducing penalty, model simple constraints, such as $x$ belonging to a simple convex set, or be a combination of both.
41
41
All $f$, $h$ and $c$ can be nonconvex.
42
42
RegularizedOptimization.jl provides a modular and extensible framework for solving \eqref{eq:nlp}, and developing novel solvers.
43
43
Currently, the following solvers are implemented:
@@ -54,7 +54,7 @@ At each iteration, a step is computed by solving a subproblem of the form \eqref
54
54
The solvers R2, R2DH and TRDH are particularly well suited to solve the subproblems, though they are general enough to solve \eqref{eq:nlp}.
55
55
All solvers are implemented in place, so re-solves incur no allocations.
56
56
To illustrate our claim of extensibility, a first version of the AL solver was implemented by an external contributor.
57
-
Furthermore, a nonsmooth penalty approach, described in [@diouane-gollier-orban-2024] is currently being developed, relying on the library’s solvers to efficiently solve its subproblems.
57
+
Furthermore, a nonsmooth penalty approach, described in [@diouane-gollier-orban-2024] is currently being developed, that relies on the libraryto efficiently solve the subproblems.
58
58
59
59
<!-- ## Requirements of the ShiftedProximalOperators.jl -->
60
60
<!---->
@@ -68,7 +68,7 @@ Furthermore, a nonsmooth penalty approach, described in [@diouane-gollier-orban-
68
68
## Model-based framework for nonsmooth methods
69
69
70
70
In Julia, \eqref{eq:nlp} can be solved using [ProximalAlgorithms.jl](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl), which implements splitting schemes and line-search–based methods [@stella-themelis-sopasakis-patrinos-2017;@themelis-stella-patrinos-2017].
71
-
Among others, the **PANOC**[@stella-themelis-sopasakis-patrinos-2017] solver takes a step along a direction $d$, which depends on the L-BFGS Quasi-Newton approximation of $f$, followed by proximal steps on $h$.
71
+
Among others, the **PANOC**[@stella-themelis-sopasakis-patrinos-2017] solver takes a step along a direction $d$, which depends on the L-BFGS quasi-Newton approximation of $f$, followed by proximal steps on $h$.
72
72
73
73
By contrast, [RegularizedOptimization.jl](https://github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl) focuses on model-based trust-region and quadratic regularization methods, which typically require fewer evaluations of $f$ and its gradient than first-order line search methods, at the expense of more evaluations of proximal operators [@aravkin-baraldi-orban-2022].
74
74
However, each proximal computation is inexpensive for numerous commonly used choices of $h$, such as separable penalties and bound constraints, so that the overall approach is efficient for large-scale problems.
@@ -85,14 +85,14 @@ Given $f$ and $h$, the companion package [RegularizedProblems.jl](https://github
85
85
reg_nlp =RegularizedNLPModel(f, h)
86
86
```
87
87
88
-
They can also be paired into a *Regularized Nonlinear LeastSquares Model* if $f(x) = \tfrac{1}{2} \|F(x)\|^2$ for some residual $F: \mathbb{R}^n \to \mathbb{R}^m$, in the case of the **LM** and **LMTR** solvers.
88
+
They can also be paired into a *Regularized Nonlinear Least-Squares Model* if $f(x) = \tfrac{1}{2} \|F(x)\|^2$ for some residual $F: \mathbb{R}^n \to \mathbb{R}^m$, in the case of the **LM** and **LMTR** solvers.
89
89
90
90
```julia
91
91
reg_nls =RegularizedNLSModel(F, h)
92
92
```
93
93
94
94
RegularizedProblems.jl also provides a set of instances commonly used in data science and in nonsmooth optimization, where several choices of $f$ can be paired with various regularizers.
95
-
This design makes for a convenient source of reproducible problem instances for benchmarking the solvers in [RegularizedOptimization.jl](https://www.github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl).
95
+
This design makes for a convenient source of problem instances for benchmarking the solvers in [RegularizedOptimization.jl](https://www.github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl).
96
96
97
97
## Support for both exact and approximate Hessian
98
98
@@ -105,7 +105,7 @@ This design allows solvers to exploit second-order information without explicitl
105
105
106
106
We illustrate the capabilities of [RegularizedOptimization.jl](https://github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl) on a Support Vector Machine (SVM) model with a $\ell_{1/2}^{1/2}$ penalty for image classification [@aravkin-baraldi-orban-2024].
107
107
108
-
Below is a condensed example showing how to define and solve an SVM problem, and perform a solve followed by a re-solve:
108
+
Below is a condensed example showing how to define and solve the problem, and perform a solve followed by a re-solve:
109
109
110
110
```julia
111
111
using LinearAlgebra, Random, ProximalOperators
@@ -139,9 +139,9 @@ For the **LM** and **LMTR** solvers, $\#\nabla f$ counts the number of Jacobian
139
139
All methods successfully reduced the optimality measure below the specified tolerance of $10^{-4}$, and thus converged to an approximate first-order stationary point.
140
140
Note that the final objective values differ due to the nonconvexity of the problem.
141
141
142
-
**R2N** is the fastest, requiring the fewest gradient evaluations.
142
+
**R2N** is the fastest in terms of time and number of gradient evaluations.
143
143
However, it requires more proximal evaluations, but these are inexpensive.
144
-
**LMTR** and **LM** require the fewest function evaluations, but incur many Jacobian–vector products, and are the slowest.
144
+
**LMTR** and **LM** require the fewest function evaluations, but incur many Jacobian–vector products, and are the slowest in terms of time.
145
145
146
146
Ongoing research aims to reduce the number of proximal evaluations.
0 commit comments