document new callbacks

Author: casperdcl

CHANGELOG.md

@@ -1,6 +1,9 @@
 * XX.X.X
   - New features:
     - LSQR algorithm added to the CIL algorithm class (#1975)
+    - Callback improvements:
+      - Added `Callback.interval` and `Callback.skip_iteration(Algorithm)` convenience methods (#1909)
+      - Added `TimingCallback` and `CSVCallback`
   - Bug fixes:
     - `CentreOfRotationCorrector.image_sharpness` data is now correctly smoothed to reduce aliasing artefacts and improve robustness. (#2202)
     - `PaganinProcessor` now correctly applies scaling with magnification for cone-beam geometry (#2225)

docs/source/optimisation.rst

@@ -101,7 +101,7 @@ SIRT
 
 ISTA/PGD
 --------
-The Iterative Soft Thresholding Algorithm (ISTA) is also known as Proximal Gradient Descent (PGD). Note that in CIL, :ref:`PGD<ISTA>` is an alias of `ISTA`. 
+The Iterative Soft Thresholding Algorithm (ISTA) is also known as Proximal Gradient Descent (PGD). Note that in CIL, :ref:`PGD<ISTA>` is an alias of `ISTA`.
 
 .. _ISTA:
 .. autoclass:: cil.optimisation.algorithms.ISTA
@@ -111,7 +111,7 @@ The Iterative Soft Thresholding Algorithm (ISTA) is also known as Proximal Gradi
 
 FISTA
 -----
-The Fast Iterative Soft Thresholding Algorithm (FISTA). 
+The Fast Iterative Soft Thresholding Algorithm (FISTA).
 
 .. _FISTA:
 .. autoclass:: cil.optimisation.algorithms.FISTA
@@ -120,18 +120,18 @@ The Fast Iterative Soft Thresholding Algorithm (FISTA).
 
 APGD
 -----
-The Accelerated Proximal Gradient Descent Algorithm (APGD). This is an extension of the PGD/ISTA algorithm allowing you to either use a constant momemtum or a momentum that is updated at each iteration. 
+The Accelerated Proximal Gradient Descent Algorithm (APGD). This is an extension of the PGD/ISTA algorithm allowing you to either use a constant momemtum or a momentum that is updated at each iteration.
 
 .. autoclass:: cil.optimisation.algorithms.APGD
    :members:
    :inherited-members: run, update_objective_interval
 
 Current options for are based on the scalar momentum, with base class:
 
-.. autoclass:: cil.optimisation.algorithms.APGD.ScalarMomentumCoefficient 
+.. autoclass:: cil.optimisation.algorithms.APGD.ScalarMomentumCoefficient
    :members:
 
-Implemented examples are: 
+Implemented examples are:
 
 .. autoclass:: cil.optimisation.algorithms.APGD.ConstantMomentum
    :members:
@@ -164,25 +164,25 @@ PD3O
 Algorithms (Stochastic)
 ========================
 
-Consider optimisation problems that take the form of a separable sum: 
+Consider optimisation problems that take the form of a separable sum:
 
 .. math:: \min_{x} f(x)+g(x) = \min_{x} \sum_{i=0}^{n-1} f_{i}(x) + g(x) = \min_{x} (f_{0}(x) + f_{1}(x) + ... + f_{n-1}(x))+g(x)
 
-where :math:`n` is the number of functions.  Where there is a large number of :math:`f_i` or their gradients are expensive to calculate, stochastic optimisation methods could prove more efficient.   
-There is a growing range of Stochastic optimisation algorithms available with potential benefits of faster convergence in number of iterations or in computational cost. 
-This is an area of continued development for CIL and, depending on  the properties of the :math:`f_i` and the regulariser :math:`g`, there is a range of different options for the user. 
+where :math:`n` is the number of functions.  Where there is a large number of :math:`f_i` or their gradients are expensive to calculate, stochastic optimisation methods could prove more efficient.
+There is a growing range of Stochastic optimisation algorithms available with potential benefits of faster convergence in number of iterations or in computational cost.
+This is an area of continued development for CIL and, depending on  the properties of the :math:`f_i` and the regulariser :math:`g`, there is a range of different options for the user.
 
 
 
 SPDHG
 -----
-Stochastic Primal Dual Hybrid Gradient (SPDHG) is a stochastic version of PDHG and deals with optimisation problems of the form: 
-    
+Stochastic Primal Dual Hybrid Gradient (SPDHG) is a stochastic version of PDHG and deals with optimisation problems of the form:
+
     .. math::
-    
+
       \min_{x} f(Kx) + g(x) = \min_{x} \sum f_i(K_i x) + g(x)
 
-where :math:`f_i` and the regulariser :math:`g` need only be proper, convex and lower semi-continuous ( i.e. do not need to be differentiable). 
+where :math:`f_i` and the regulariser :math:`g` need only be proper, convex and lower semi-continuous ( i.e. do not need to be differentiable).
 Each iteration considers just one index of the sum, potentially reducing computational cost. For more examples see our [user notebooks]( https://github.com/vais-ral/CIL-Demos/blob/master/Tomography/Simulated/Single%20Channel/PDHG_vs_SPDHG.py).
 
 
@@ -194,23 +194,23 @@ Each iteration considers just one index of the sum, potentially reducing computa
 Approximate gradient methods
 ----------------------------------
 
-Alternatively, consider that, in addition to the functions :math:`f_i` and the regulariser :math:`g` being proper, convex and lower semi-continuous, the :math:`f_i` are differentiable. In this case we consider stochastic methods that replace a gradient calculation in a deterministic algorithm with a, potentially cheaper to calculate, approximate gradient. 
+Alternatively, consider that, in addition to the functions :math:`f_i` and the regulariser :math:`g` being proper, convex and lower semi-continuous, the :math:`f_i` are differentiable. In this case we consider stochastic methods that replace a gradient calculation in a deterministic algorithm with a, potentially cheaper to calculate, approximate gradient.
 For example, when :math:`g(x)=0`, the standard Gradient Descent algorithm utilises iterations of the form
 
    .. math::
       x_{k+1}=x_k-\alpha \nabla f(x_k) =x_k-\alpha \sum_{i=0}^{n-1}\nabla f_i(x_k).
-:math:`\nabla f(x_k)=\sum_{i=0}^{n-1}\nabla f_i(x_k)` with :math:`n \nabla f_i(x_k)`, for an index :math:`i` which changes each iteration, leads to the well known stochastic gradient descent algorithm. 
+:math:`\nabla f(x_k)=\sum_{i=0}^{n-1}\nabla f_i(x_k)` with :math:`n \nabla f_i(x_k)`, for an index :math:`i` which changes each iteration, leads to the well known stochastic gradient descent algorithm.
 
 
 
-Replacing, :math:`\nabla f(x_k)=\sum_{i=0}^{n-1}\nabla f_i(x_k)` with :math:`n \nabla f_i(x_k)`, for an index :math:`i` which changes each iteration, leads to the well known stochastic gradient descent algorithm. 
+Replacing, :math:`\nabla f(x_k)=\sum_{i=0}^{n-1}\nabla f_i(x_k)` with :math:`n \nabla f_i(x_k)`, for an index :math:`i` which changes each iteration, leads to the well known stochastic gradient descent algorithm.
 
-In addition, if :math:`g(x)\neq 0` and has a calculable proximal ( need not be differentiable) one can consider ISTA iterations: 
+In addition, if :math:`g(x)\neq 0` and has a calculable proximal ( need not be differentiable) one can consider ISTA iterations:
 
    .. math::
          x_{k+1}=prox_{\alpha g}(x_k-\alpha \nabla f(x_k) )=prox_{\alpha g}(x_k-\alpha \sum_{i=0}^{n-1}\nabla f_i(x_k))
 
-and again replacing :math:`\nabla f(x_k)=\sum_{i=0}^{n-1}\nabla f_i(x_k)` with an approximate gradient. 
+and again replacing :math:`\nabla f(x_k)=\sum_{i=0}^{n-1}\nabla f_i(x_k)` with an approximate gradient.
 
 In a similar way, plugging approximate gradient calculations into deterministic algorithms can lead to a range of stochastic algorithms. In the following table, the left hand column has the approximate gradient function subclass, :ref:`Approximate Gradient base class` the header row has one of CIL's deterministic optimisation algorithm and the body of the table has the resulting stochastic algorithm.
 
@@ -228,9 +228,9 @@ In a similar way, plugging approximate gradient calculations into deterministic
 | LSVRGFunction\| LSVRG | Prox-LSVRG | Acc-Prox-LSVRG |
 +----------------+-------+------------+----------------+
 
-\*In development 
+\*In development
 
-The stochastic gradient functions can be found listed under functions in the documentation. 
+The stochastic gradient functions can be found listed under functions in the documentation.
 
 Stochastic Gradient Descent Example
 ----------------------------------
@@ -239,52 +239,52 @@ The below is an example of Stochastic Gradient Descent built of the SGFunction a
 .. code-block :: python
 
    from cil.optimisation.utilities import Sampler
-   from cil.optimisation.algorithms import GD 
+   from cil.optimisation.algorithms import GD
    from cil.optimisation.functions import LeastSquares, SGFunction
    from cil.utilities import dataexample
    from cil.plugins.astra.operators import ProjectionOperator
-   
-   # get the data  
+
+   # get the data
    data = dataexample.SIMULATED_PARALLEL_BEAM_DATA.get()
    data.reorder('astra')
    data = data.get_slice(vertical='centre')
 
-   # create the geometries 
-   ag = data.geometry 
+   # create the geometries
+   ag = data.geometry
    ig = ag.get_ImageGeometry()
 
    # partition the data and build the projectors
-   n_subsets = 10 
+   n_subsets = 10
    partitioned_data = data.partition(n_subsets, 'sequential')
    A_partitioned = ProjectionOperator(ig, partitioned_data.geometry, device = "cpu")
 
-   # create the list of functions for the stochastic sum 
+   # create the list of functions for the stochastic sum
    list_of_functions = [LeastSquares(Ai, b=bi) for Ai,bi in zip(A_partitioned, partitioned_data)]
 
-   #define the sampler and the stochastic gradient function 
+   #define the sampler and the stochastic gradient function
    sampler = Sampler.staggered(len(list_of_functions), stride=2)
-   f = SGFunction(list_of_functions, sampler=sampler)  
-   
-   #set up and run the gradient descent algorithm 
+   f = SGFunction(list_of_functions, sampler=sampler)
+
+   #set up and run the gradient descent algorithm
    alg = GD(initial=ig.allocate(0), objective_function=f, step_size=1/f.L)
    alg.run(300)
 
 
 Note
 ----
- All the approximate gradients written in CIL are of a similar order of magnitude to the full gradient calculation. For example, in the :code:`SGFunction` we approximate the full gradient by :math:`n\nabla f_i` for an index :math:`i` given by the sampler. 
+ All the approximate gradients written in CIL are of a similar order of magnitude to the full gradient calculation. For example, in the :code:`SGFunction` we approximate the full gradient by :math:`n\nabla f_i` for an index :math:`i` given by the sampler.
  The multiplication by :math:`n` is a choice to more easily allow comparisons between stochastic and non-stochastic methods and between stochastic methods with varying numbers of subsets.
  The multiplication ensures that the (SAGA, SGD, and SVRG  and LSVRG) approximate gradients are an unbiased estimator of the full gradient ie :math:`\mathbb{E}\left[\tilde\nabla f(x)\right] =\nabla f(x)`.
-  This has an implication when choosing step sizes. For example, a suitable step size for GD with a SGFunction could be 
-  :math:`\propto 1/(L_{max}*n)`, where :math:`L_{max}` is the largest Lipschitz constant of the list of functions in the SGFunction and the additional factor of  :math:`n` reflects this multiplication by  :math:`n` in the approximate gradient. 
+  This has an implication when choosing step sizes. For example, a suitable step size for GD with a SGFunction could be
+  :math:`\propto 1/(L_{max}*n)`, where :math:`L_{max}` is the largest Lipschitz constant of the list of functions in the SGFunction and the additional factor of  :math:`n` reflects this multiplication by  :math:`n` in the approximate gradient.
+
 
-  
 Memory requirements
 -------------------
 Note that the approximate gradient methods have different memory requirements:
 + The `SGFunction` has the same requirements as a `SumFunction`, so no increased memory usage
-+ `SAGFunction` and `SAGAFunction` both store `n+3` times the image size in memory to store the last calculated gradient for each function in the sum and for intermediary calculations. 
-+ `SVRGFunction` and `LSVRGFunction` with the default `store_gradients = False` store 4 times the image size in memory, including the "snapshot" point and gradient. If `store_gradients = True`, some computational effort is saved, at the expensive of stored memory `n+4` times the image size.  
++ `SAGFunction` and `SAGAFunction` both store `n+3` times the image size in memory to store the last calculated gradient for each function in the sum and for intermediary calculations.
++ `SVRGFunction` and `LSVRGFunction` with the default `store_gradients = False` store 4 times the image size in memory, including the "snapshot" point and gradient. If `store_gradients = True`, some computational effort is saved, at the expensive of stored memory `n+4` times the image size.
 
 
 Operators
@@ -542,55 +542,55 @@ Total variation
    :members:
    :inherited-members:
 
-Function of Absolute Value 
+Function of Absolute Value
 --------------------------
 
 .. autoclass:: cil.optimisation.functions.FunctionOfAbs
    :members:
    :inherited-members:
 
 
-Approximate Gradient base class 
+Approximate Gradient base class
 --------------------------------
 
-.. autoclass:: cil.optimisation.functions.ApproximateGradientSumFunction 
+.. autoclass:: cil.optimisation.functions.ApproximateGradientSumFunction
    :members:
    :inherited-members:
-   
 
-Stochastic Gradient function 
+
+Stochastic Gradient function
 -----------------------------
 
-.. autoclass:: cil.optimisation.functions.SGFunction 
+.. autoclass:: cil.optimisation.functions.SGFunction
    :members:
    :inherited-members:
 
 SAG function
 -------------
 
-.. autoclass:: cil.optimisation.functions.SAGFunction 
+.. autoclass:: cil.optimisation.functions.SAGFunction
    :members:
    :inherited-members:
 
 SAGA function
 --------------
 
-.. autoclass:: cil.optimisation.functions.SAGAFunction 
+.. autoclass:: cil.optimisation.functions.SAGAFunction
    :members:
    :inherited-members:
 
 
 
-Stochastic Variance Reduced Gradient Function 
+Stochastic Variance Reduced Gradient Function
 ----------------------------------------------
-.. autoclass:: cil.optimisation.functions.SVRGFunction 
+.. autoclass:: cil.optimisation.functions.SVRGFunction
    :members:
    :inherited-members:
 
 
-Loopless Stochastic Variance Reduced Gradient Function 
+Loopless Stochastic Variance Reduced Gradient Function
 ----------------------------------------------
-.. autoclass:: cil.optimisation.functions.LSVRGFunction 
+.. autoclass:: cil.optimisation.functions.LSVRGFunction
    :members:
    :inherited-members:
 
@@ -625,12 +625,12 @@ For ease of use we provide the following static methods in `cil.optimisation.uti
 They will all instantiate a Sampler defined in the following class:
 
 .. autoclass:: cil.optimisation.utilities.Sampler
-   
+
 
 The random samplers are instantiated from a random sampling class which is a child class of  `cil.optimisation.utilities.sampler` and provides options for sampling with and without replacement:
 
 .. autoclass:: cil.optimisation.utilities.SamplerRandom
-   
+
 
 Callbacks
 ---------
@@ -657,6 +657,12 @@ Built-in callbacks include:
 .. autoclass:: cil.optimisation.utilities.callbacks.LogfileCallback
    :members:
 
+.. autoclass:: cil.optimisation.utilities.callbacks.CSVCallback
+   :members:
+
+.. autoclass:: cil.optimisation.utilities.callbacks.TimingCallback
+   :members:
+
 .. autoclass:: cil.optimisation.utilities.callbacks.EarlyStoppingObjectiveValue
    :members:
 
@@ -700,9 +706,9 @@ In each iteration of the :code:`TestAlgo`, the objective :math:`x` is reduced by
     15%|███                 | 3/20 [00:00<00:00, 11770.73it/s, objective=3.05e-5]
 
 
-Step size methods 
+Step size methods
 ------------------
-A step size method is a class which acts on an algorithm and can be passed to  `cil.optimisation.algorithm.GD`, `cil.optimisation.algorithm.ISTA`  `cil.optimisation.algorithm.FISTA` and it's method `get_step_size` is called after the calculation of the gradient before the gradient descent step is taken. It outputs a float value to be used as the step-size. 
+A step size method is a class which acts on an algorithm and can be passed to  `cil.optimisation.algorithm.GD`, `cil.optimisation.algorithm.ISTA`  `cil.optimisation.algorithm.FISTA` and it's method `get_step_size` is called after the calculation of the gradient before the gradient descent step is taken. It outputs a float value to be used as the step-size.
 
 Currently in CIL we have a base class:
 
@@ -724,7 +730,7 @@ We also have a number of example classes:
 
 Preconditioners
 ----------------
-A preconditioner is a class which acts on an algorithm and can be passed to  `cil.optimisation.algorithm.GD`, `cil.optimisation.algorithm.ISTA` or `cil.optimisation.algorithm.FISTA` and it's method `apply` is called after the calculation of the gradient before the gradient descent step is taken. It modifies and returns a passed `gradient`. 
+A preconditioner is a class which acts on an algorithm and can be passed to  `cil.optimisation.algorithm.GD`, `cil.optimisation.algorithm.ISTA` or `cil.optimisation.algorithm.FISTA` and it's method `apply` is called after the calculation of the gradient before the gradient descent step is taken. It modifies and returns a passed `gradient`.
 
 Currently in CIL we have a base class: