Skip to content

Commit b06581b

Browse files
MohamedLaghdafHABIBOULLAHMohamedLaghdafHABIBOULLAH
committed
add first results on fh
1 parent 3c68da0 commit b06581b

2 files changed

Lines changed: 59 additions & 45 deletions

File tree

‎paper/examples/example2.jl‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ using ADNLPModels, NLPModels, NLPModelsModifiers, RegularizedOptimization, Regul
88
model, _, _ = RegularizedProblems.fh_model()
99

1010
# Define the Hessian approximation
11-
f = LBFGSModel(fh_model)
11+
f = LBFGSModel(model)
1212

1313
# Define the nonsmooth regularizer (L1 norm)
1414
λ = 0.1

‎paper/paper.md‎

Lines changed: 58 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -74,71 +74,69 @@ Finally, nonsmooth terms $h$ can be modeled using [ProximalOperators.jl](https:/
7474

7575
This modularity makes it easy to benchmark existing solvers available in the repository [@diouane-habiboullah-orban-2024], [@aravkin-baraldi-orban-2022], [@aravkin-baraldi-orban-2024], and [@leconte-orban-2023-2].
7676

77-
## Support for inexact subproblem solves
77+
## Support for Hessians
7878

79-
Solvers in **RegularizedOptimization.jl** allow inexact resolution of trust-region and quadratic-regularized subproblems using first-order that are implemented in the package itself such as the quadratic regularization method R2[@aravkin-baraldi-orban-2022] and R2DH[@diouane-habiboullah-orban-2024] with trust-region variants TRDH[@leconte-orban-2023-2].
79+
In contrast to first-order methods package like [ProximalAlgorithms.jl](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl), **RegularizedOptimization.jl** enables the use of second-order information, which can significantly improve convergence rates, especially for ill-conditioned problems.
80+
A way to use Hessians is via automatic differentiation tools such as [ADNLPModels.jl](https://github.com/JuliaSmoothOptimizers/ADNLPModels.jl).
8081

81-
This is crucial for large-scale problems where exact subproblem solutions are prohibitive.
82+
## Requirements of the ShiftedProximalOperators.jl package
8283

83-
## Support for Hessians as Linear Operators
84+
The nonsmooth part $h$ must have a computable proximal mapping, defined as
85+
$$\text{prox}_{h}(v) = \underset{x \in \mathbb{R}^n}{\arg\min} \left( h(x) + \frac{1}{2} \|x - v\|^2 \right).$$
86+
This requirement is satisfied by a wide range of nonsmooth functions commonly used in practice, such as the $\ell_1$ norm, the $\ell_0$ "norm", indicator functions of convex sets, and group sparsity-inducing norms.
87+
The package [ProximalOperators.jl](https://www.github.com/FirstOrder/ProximalOperators.jl) provides a comprehensive collection of such functions, along with their proximal mappings.
88+
The main difference between the proximal operators implemented in
89+
[ProximalOperators.jl](https://github.com/JuliaFirstOrder/ProximalOperators.jl)
90+
is that those implemented here involve a translation of the nonsmooth term.
91+
Specifically, this package considers proximal operators defined as
92+
$$
93+
argmin \, { \tfrac{1}{2} ‖t - q‖₂² + ν h(x + s + t) + χ(s + t; ΔB) | t ∈ ℝⁿ },
94+
$$
95+
where q is given, x and s are fixed shifts, h is the nonsmooth term with respect
96+
to which we are computing the proximal operator, and χ(.; ΔB) is the indicator of
97+
a ball of radius Δ defined by a certain norm.
8498

85-
The second-order methods in **RegularizedOptimization.jl** can use Hessian approximations represented as linear operators via [LinearOperators.jl](https://github.com/JuliaSmoothOptimizers/LinearOperators.jl).
86-
Explicitly forming Hessians as dense or sparse matrices is often prohibitively expensive, both computationally and in terms of memory, especially in high-dimensional settings.
87-
In contrast, many problems admit efficient implementations of Hessian–vector or Jacobian–vector products, either through automatic differentiation tools or limited-memory quasi-Newton updates, making the linear-operator approach more scalable and practical.
99+
## Testing and documentation
88100

89-
## In-place methods
101+
The package includes a comprehensive suite of unit tests that cover all functionalities, ensuring reliability and correctness.
102+
Extensive documentation is provided, including a user guide, API reference, and examples to help users get started quickly.
103+
Aqua.jl is used to test the package dependencies.
104+
Documentation is built using Documenter.jl.
90105

91-
All solvers in **RegularizedOptimization.jl** are implemented in an in-place fashion, minimizing memory allocations during the resolution process.
106+
## Hyperparameter tuning
92107

93-
# Examples
108+
The solvers in **RegularizedOptimization.jl** do not require extensive hyperparameter tuning.
94109

95-
We consider two examples where the smooth part $f$ is nonconvex and the nonsmooth part $h$ is either the $\ell_0$ or $\ell_1$ norm.
110+
## Non-monotone strategies
96111

97-
A first example addresses an image recognition task using a support vector machine (SVM) similar to those in [@aravkin-baraldi-orban-2022] and [@diouane-habiboullah-orban-2024].
98-
The formulation is
99-
$$
100-
\min_{x \in \mathbb{R}^n} \ \tfrac{1}{2} \|\mathbf{1} - \tanh(b \odot \langle A, x \rangle)\|^2 + \lambda \|x\|_0,
101-
$$
102-
where $\lambda = 10^{-1}$ and $A \in \mathbb{R}^{m \times n}$, with $n = 784$ representing the vectorized size of each image and $m = 13{,}007$ is the number of images in the training dataset.
112+
The solvers in **RegularizedOptimization.jl** implement non-monotone strategies to accept trial points, which can enhance convergence properties.
103113

104-
```julia
105-
using LinearAlgebra, Random
106-
using ProximalOperators
107-
using NLPModels, NLPModelsModifiers, RegularizedProblems, RegularizedOptimization
108-
using MLDatasets
114+
## Application studies
109115

110-
random_seed = 1234
111-
Random.seed!(random_seed)
116+
The package can be applied to the exact penality work by [@diouane-gollier-orban-2024] that addresses a problem where the model of the nonsmooth part is different from the function $h$.
117+
This is not covered in the current version of the competitive package [ProximalAlgorithms.jl](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl).
112118

113-
# Load MNIST from MLDatasets
114-
imgs, labels = MLDatasets.MNIST.traindata()
119+
## Support for inexact subproblem solves
115120

116-
# Use RegularizedProblems' preprocessing
117-
A, b = RegularizedProblems.generate_data(imgs, labels, (1, 7), false)
121+
Solvers in **RegularizedOptimization.jl** allow inexact resolution of trust-region and quadratic-regularized subproblems using first-order that are implemented in the package itself such as the quadratic regularization method R2[@aravkin-baraldi-orban-2022] and R2DH[@diouane-habiboullah-orban-2024] with trust-region variants TRDH[@leconte-orban-2023-2].
118122

119-
# Build the models
120-
model, _, _ = RegularizedProblems.svm_model(A, b)
123+
This is crucial for large-scale problems where exact subproblem solutions are prohibitive.
121124

122-
# Define the Hessian approximation
123-
f = LBFGSModel(model)
125+
## Support for Hessians as Linear Operators
124126

125-
# Define the nonsmooth regularizer (L0 norm)
126-
λ = 1.0e-1
127-
h = NormL0(λ)
127+
The second-order methods in **RegularizedOptimization.jl** can use Hessian approximations represented as linear operators via [LinearOperators.jl](https://github.com/JuliaSmoothOptimizers/LinearOperators.jl).
128+
Explicitly forming Hessians as dense or sparse matrices is often prohibitively expensive, both computationally and in terms of memory, especially in high-dimensional settings.
129+
In contrast, many problems admit efficient implementations of Hessian–vector or Jacobian–vector products, either through automatic differentiation tools or limited-memory quasi-Newton updates, making the linear-operator approach more scalable and practical.
128130

129-
# Define the regularized NLP model
130-
reg_nlp = RegularizedNLPModel(f, h)
131+
## In-place methods
131132

132-
# Choose a solver (R2DH) and execution statistics tracker
133-
solver_r2dh= R2DHSolver(reg_nlp)
134-
stats = RegularizedExecutionStats(reg_nlp)
133+
All solvers in **RegularizedOptimization.jl** are implemented in an in-place fashion, minimizing memory allocations during the resolution process.
135134

136-
# Solve the problem
137-
solve!(solver_r2dh, reg_nlp, stats, x = f.meta.x0, σk = 1e-6, atol = 2e-5, rtol = 2e-5, verbose = 1)
135+
# Examples
138136

139-
```
137+
We consider two examples where the smooth part $f$ is nonconvex and the nonsmooth part $h$ is either the $\ell_0$ or $\ell_1$ norm.
140138

141-
Another example is the FitzHugh-Nagumo inverse problem with an $\ell_1$ penalty, as described in [@aravkin-baraldi-orban-2022] and [@aravkin-baraldi-orban-2024].
139+
A first example is the FitzHugh-Nagumo inverse problem with an $\ell_1$ penalty, as described in [@aravkin-baraldi-orban-2022] and [@aravkin-baraldi-orban-2024].
142140

143141
```julia
144142
using LinearAlgebra
@@ -167,6 +165,22 @@ stats = RegularizedExecutionStats(reg_nlp)
167165
solve!(solver_tr, reg_nlp, stats, x = f.meta.x0, atol = 1e-3, rtol = 1e-4, verbose = 10, ν = 1.0e+2)
168166
```
169167

168+
````
169+
=== Comparaison PANOC vs TR (FH_smooth_term) ===
170+
PANOC :
171+
itérations = 81
172+
# f évaluations = 188
173+
# ∇f évaluations = 188
174+
# prox appels (g) = 107
175+
solution (≈) = [-0.0, 0.19071674721048656, 1.037084478194805, -0.0, -0.0]
176+
177+
TR :
178+
statut = first_order
179+
# f évaluations = 65
180+
# ∇f évaluations = 52
181+
solution (≈) = [0.0, 0.1910326406395867, 1.0357773976471938, 0.0, 0.0]
182+
````
183+
170184
# Acknowledgements
171185

172186
Mohamed Laghdaf Habiboullah is supported by an excellence FRQNT grant,

0 commit comments

Comments
 (0)