Convergence logging

[mileslucas]

Feature Request

Being able to log progress without having a set number of iterations would be great. Currently AbstractMCMC is using ProgressLogging as a backend and they've just added convergence sampling.

This is currently implemented in ProgressMeters.jl. I really like how it spits out the total time information in addition to just updating current values vs. threshold.

Example Usage

Something like this:

cur = 0
goal = 10
@withprogresslogging while true
    @logprogress cur goal
    cur += rand()
    cur < goal || break
end

[tkf]

It sounds like a good addition. I think a straightforward implementation would be to add a new type like ProgressLogging.Progress. Something like Convergence? Descent/Ascent?

I think the surface syntax requires a separate macro; e.g., @logconvergence, @logdescent, @logascent.

[c42f]

Something like Convergence?

This sounds good to me.

Convergence logging may also be interesting to TensorBoardLogger users and devs who at the moment have their own custom wrapper types and dispatch rules. @PhilipVinc do you have any thoughts?

[mileslucas]

In ProgressMeter they use threshold. So

@logthreshold cur goal

has the "least surprising" behavior to me, although I see the keyword threshold is tied up with the @progress macro, so it may complicate things here...

[tkf]

How about Descent/Ascent? Do we want to support only the minimization case? Optim has maximize function so I thought it might make sense to have different directions. Another possibility is to have a boolean flag in Convergence struct.

[c42f]

My thought about threshold is that there's a lot of different ways to measure convergence and it would be nice to have a more general wrapper for convergence measures. I guess Optim.jl is a good place to look? For example here https://julianlsolvers.github.io/Optim.jl/stable/#examples/generated/ipnewton_basics/ they list convergence criteria in the output reports such as

 * Status: success

 * Candidate solution
    Minimizer: [1.00e+00, 1.00e+00]
    Minimum:   5.998937e-19

 * Found with
    Algorithm:     Interior Point Newton
    Initial Point: [0.00e+00, 0.00e+00]

 * Convergence measures
    |x - x'|               = 1.50e-09 ≰ 0.0e+00
    |x - x'|/|x'|          = 1.50e-09 ≰ 0.0e+00
    |f(x) - f(x')|         = 1.80e-18 ≰ 0.0e+00
    |f(x) - f(x')|/|f(x')| = 3.00e+00 ≰ 0.0e+00
    |g(x)|                 = 7.92e-09 ≤ 1.0e-08

 * Work counters
    Seconds run:   0  (vs limit Inf)
    Iterations:    34
    f(x) calls:    63
    ∇f(x) calls:   63

@pkofod if we were to have a general solution for logging optimization progress, what would you need it to do?

[c42f]

See also JuliaNLSolvers/Optim.jl#442

[tkf]

We can have multiple Descent instances with different IDs. So the progress monitor UI should be able to show different convergence criteria if we only define one or two convergence message types for scalar quantities.

[c42f]

We can have multiple Descent instances with different IDs

On the other hand, it's a single computational process with multiple convergence criteria which apply. So I would have thought the UI should present it as a single message. In that case we should also emit the progress as a single message (or the UI will need to try to figure out how to group them).

[tkf]

If they share the same parent ID, I suppose it's not too hard for the UI to figure out the relationships? Though it's appealing that we can reduce shouldlog overhead this way. I guess this may be one of the cases where "vectorized API" is OK due to the performance reason.

[tkf]

Hmm... I wonder if it is better to somehow combine this to table log monitor JuliaLogging/TerminalLoggers.jl#8

The idea is that each "column" can have independent threshold value, descent/ascent flag, and leave-trace-or-not flag.

[PhilipVinc]

@c42f TensorBoard does not natively support logging convergence information like this, but I guess it would be a nice (and easy) addition.

The way TB.jl is structured, we parse one log message at a time. It would surely makes things easier if you log just one message with all the metrics in it.

[c42f]

The way TB.jl is structured, we parse one log message at a time. It would surely makes things easier if you log just one message with all the metrics in it.

So if you're tracking model accuracy or some other measures, the user needs to accumulate that each iteration and pass the resulting array to the logger?

[PhilipVinc]

So if you're tracking model accuracy or some other measures, the user needs to accumulate that each iteration and pass the resulting array to the logger?

No, sorry, I simply meant that we do not have a special-cased convergence object. You can simply log at every iteration your loss function and those will all be displayed. For example

struct ConvergenceLog
 L1
 L2
 weirdnorm
end

with_logger(TBLogger()) begin
  for i=1:100
    loss = ConvergenceLog(exp(-i), exp(-2i), rand())
    @info "" loss
  end
end

will by default create 3 plots, one for L1, one for L2 and one for weird norm. Like anything else.

[tkf]

@PhilipVinc You need to pass log_step_increment=0 to log multiple objects in one iteration, right? It may be what @c42f meant by "accumulate."

"Vectorized" message type is indeed more TensorBoardLogger.jl friendly. Although TensorBoardLogger.jl needs to understand that it is an "interface type" rather than a "data type" (ref StructTypes.jl). More concretely, the message type would be something like

struct Convergence
    names::Vector{String}
    values::Vector{Float64}
    thresholds::Vector{Float64}
    descending::Vector{Bool}  # or maybe directions::Vector{Int8} that contains -1, 0, or 1 (0: no threshold)
    id::UUID
    parentid::UUID
    done::Bool
    step::Int
end

We could make it a subtype of AbstractDict{String,Float64} s.t. (c::Convergence)[name] == c.values[findfirst(name, c.names)] so that it works without a special handling in TensorBoardLogger.jl. But it's strange that most of semantic data is ignored at the interface type level.

(Edit: step::Int added)