Skip to content

Commit ee28d76

Browse files
Address editor's comments
1 parent 83e66ad commit ee28d76

2 files changed

Lines changed: 19 additions & 8 deletions

File tree

paper/paper.bib

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,3 +159,14 @@ @article{ eckstein-bertsekas-1992
159159
publisher = {Springer},
160160
doi = {10.1007/BF01581204}
161161
}
162+
163+
@techreport{allaire-le-digabel-orban-2025,
164+
title = {An inexact modified quasi-Newton method for nonsmooth regularized optimization},
165+
author = {Allaire, Nathan and Le Digabel, S{\'e}bastien and Orban, Dominique},
166+
institution = {GERAD},
167+
type = {Cahier},
168+
number = {G-2025-73},
169+
year = {2025},
170+
address = {Montr{\'e}al, Canada},
171+
url = {10.13140/RG.2.2.32728.97288}
172+
}

paper/paper.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -50,11 +50,11 @@ Currently, the following solvers are implemented:
5050
All solvers rely on first derivatives of $f$ and $c$, and optionally on their second derivatives in the form of Hessian-vector products.
5151
If second derivatives are not available, quasi-Newton approximations can be used.
5252
In addition, the proximal mapping of the nonsmooth part $h$, or adequate models thereof, must be evaluated.
53-
At each iteration, a step is computed by solving a subproblem of the form \eqref{eq:nlp} inexactly, in which $f$, $h$, and $c$ are replaced with appropriate models about the current iterate.
53+
At each iteration, a step is computed by solving a subproblem of the form \eqref{eq:nlp} inexactly, in which $f$, $h$, and $c$ are replaced with appropriate models around the current iterate.
5454
The solvers R2, R2DH and TRDH are particularly well suited to solve the subproblems, though they are general enough to solve \eqref{eq:nlp}.
55-
All solvers are implemented in place, so re-solves incur no allocations.
55+
All solvers are allocation-free, so re-solves incur no additional allocations.
5656
To illustrate our claim of extensibility, a first version of the AL solver was implemented by an external contributor.
57-
Furthermore, a nonsmooth penalty approach, described in [@diouane-gollier-orban-2024] is currently being developed, that relies on the library to efficiently solve the subproblems.
57+
Furthermore, a nonsmooth penalty approach, described in [@diouane-gollier-orban-2024], is currently being developed, that relies on the library to efficiently solve the subproblems.
5858

5959
<!-- ## Requirements of the ShiftedProximalOperators.jl -->
6060
<!---->
@@ -85,7 +85,7 @@ Given $f$ and $h$, the companion package [RegularizedProblems.jl](https://github
8585
reg_nlp = RegularizedNLPModel(f, h)
8686
```
8787

88-
They can also be paired into a *Regularized Nonlinear Least-Squares Model* if $f(x) = \tfrac{1}{2} \|F(x)\|^2$ for some residual $F: \mathbb{R}^n \to \mathbb{R}^m$, in the case of the **LM** and **LMTR** solvers.
88+
They can also be paired into a *Regularized Nonlinear Least-Squares Model*, used by the **LM** and **LMTR** solvers, if $f(x) = \tfrac{1}{2} \|F(x)\|^2$ for some residual $F: \mathbb{R}^n \to \mathbb{R}^m$.
8989

9090
```julia
9191
reg_nls = RegularizedNLSModel(F, h)
@@ -96,7 +96,7 @@ This design makes for a convenient source of problem instances for benchmarking
9696

9797
## Support for both exact and approximate Hessian
9898

99-
In contrast with [ProximalAlgorithms.jl](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl), [RegularizedOptimization.jl](https://github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl), methods such as **R2N** and **TR** methods support exact Hessians as well as several Hessian approximations of $f$.
99+
In contrast to [ProximalAlgorithms.jl](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl), [RegularizedOptimization.jl](https://github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl), methods such as **R2N** and **TR** support exact Hessians as well as several Hessian approximations of $f$.
100100
Hessian–vector products $v \mapsto Hv$ can be obtained via automatic differentiation through [ADNLPModels.jl](https://github.com/JuliaSmoothOptimizers/ADNLPModels.jl) or implemented manually.
101101
Limited-memory and diagonal quasi-Newton approximations can be selected from [LinearOperators.jl](https://github.com/JuliaSmoothOptimizers/LinearOperators.jl).
102102
This design allows solvers to exploit second-order information without explicitly forming dense or sparse Hessians, which is often expensive in time and memory, particularly at large scale.
@@ -105,7 +105,7 @@ This design allows solvers to exploit second-order information without explicitl
105105

106106
We illustrate the capabilities of [RegularizedOptimization.jl](https://github.com/JuliaSmoothOptimizers/RegularizedOptimization.jl) on a Support Vector Machine (SVM) model with a $\ell_{1/2}^{1/2}$ penalty for image classification [@aravkin-baraldi-orban-2024].
107107

108-
Below is a condensed example showing how to define and solve the problem, and perform a solve followed by a re-solve:
108+
Below is a condensed example showing how to define the problem and perform a solve followed by a re-solve:
109109

110110
```julia
111111
using LinearAlgebra, Random, ProximalOperators
@@ -129,7 +129,7 @@ solve!(solver, reg_nlp, stats; atol=1e-5, rtol=1e-5, verbose=1, sub_kwargs=(max_
129129
We compare **TR**, **R2N**, **LM** and **LMTR** from our library on the SVM problem.
130130
Experiments were performed on macOS (arm64) on an Apple M2 (8-core) machine, using Julia 1.11.7.
131131

132-
The table reports the convergence status of each solver, the number of evaluations of $f$, the number of evaluations of $\nabla f$, the number of proximal operator evaluations, the elapsed time and the final objective value.
132+
The table reports the convergence status of each solver, the number of evaluations of $f$, the number of evaluations of $\nabla f$, the number of proximal operator evaluations, the elapsed time, and the final objective value.
133133
For TR and R2N, we use limited-memory SR1 Hessian approximations.
134134
The subproblem solver is **R2**.
135135

@@ -144,7 +144,7 @@ Note that the final objective values differ due to the nonconvexity of the probl
144144
However, it requires more proximal evaluations, but these are inexpensive.
145145
**LMTR** and **LM** require the fewest function evaluations, but incur many Jacobian–vector products, and are the slowest in terms of time.
146146

147-
Ongoing research aims to reduce the number of proximal evaluations.
147+
Ongoing research aims to reduce the number of proximal evaluations, for instance by allowing inexact proximal computations [@allaire-le-digabel-orban-2025].
148148

149149
# Acknowledgements
150150

0 commit comments

Comments
 (0)