Instability with HyperProp

[hadaev8]

I tried it in two tasks, but got nans during training, any suggestions?

[JRC1995]

I am not sure. I don't remember if I really tested LaProp based optimizer all that much. Is there any instability with other optimizers that use hypergradient descent? There could still be some bug in the repo too. Most of them are mix mash of multiple techniques, it may be also possible that some of these combinations simply don't work out, or even lead to some mathematical issues that I have missed; so if you aren't using something standard there may be issues with the combination idea itself. As for me I think I mostly did some sanity runs on hypergrad in first few epochs on a more basic optimizer. It didn't seem like there was anything obviously wrong from the changes in the lr, but it's hard to say just from that. My implementation of hypergrad is also a bit different from the original papers. Not sure if that causes any instability.

[hadaev8]

I tested HyperProp on 4 tasks, first two are fine (nvidia waveglow and resnet image classification), one crush then I use weight decay param (if I remember case right, maybe it just crush) (a fork of nvidia tacotron2) and the last task is transformer image gan.

I tested also HDQHSGDW on the last task and it seems to be fine.
If you have time for debugging, I can share code and data of the last case for reproducing issue (its colab notebook, so easy to share and run).

What optimizer you would recommend as default nobrain drop-in replacement?
I took HyperProp because it's last in the file, so I assumed its more advanced.

[JRC1995]

It depends on what you are trying to do. Research? Research on optimizers, specifically? Just applications? Or simply playing around with optimizers?

I haven't personally tried the optimizers exhaustively, so I can't really tell which synergistic combination is good to go. The whole hyperxxx series optimizers may not be even tried by anyone; especially given my implementation has differences than original hypergrad descent, plus, I think also added some extra gradient computation to handle weight decay and other things.

Generally, it's probably just better to stick with AdamW. This repo is mostly experimental and more for playing around.
With AdamW you can add lookahead, or try RAdam + Lookahead combination (some calls it Ranger). That should be decent. QHRAdam + Lookahead should be decent too. Some reported good results with AdaMod too. So AdaMod + Lookahead may be a good choice too.

This should recover AdaMod + lookahead

optimizer = DemonRanger(params=model.parameters(),
                        lr=config.lr,
                        weight_decay=config.wd,
                        betas=(0.9,0.999,0.999), # restore default AdamW betas
                        nus=(1.0,1.0), # disables QHMomentum
                        IA=False, # disables Iterate Averaging
                        rectify=False, # disables RAdam Recitification
                        AdaMod_bias_correct=False, #disables AdaMod bias corretion (not used originally)
                        use_demon=False #disables Decaying Momentum (DEMON)
                        use_gc=False #disables gradient centralization
                        amsgrad=False # disables amsgrad
                        )

Hypergradient descent is interesting conceptually. You can try and compare it. The original paper says even not-much-tuned hypergradient descent on lr is better than tuning lr in Adam and so on. But I don't think it was extensively tested on multiple tasks. But the gradients for hypergradient descent gets more complicated with AdaMod I think, and there could be some bugs in my gradient computation implementations in that case; and it's probably slow too. If you want to try that you probably have to see: HyperRangerMod

[hadaev8]

Production/hobby.
For now I on mine modification of ranger + lookahead + gc + diffgrad. But you did much greater work.
I did not found computation overhead very important (around +10%).
https://i.imgur.com/z9PD2br.png
Very noisy graph, but seems HyperProp (red) coverage much faster.

[hadaev8]

I had more tests with all optimizers (and different params) on my TTS task and for some reason, it performed much worse than my current optimizer.
Orange is my version of ranger, dark green is default adamw.

Also, I noticed nan loss on validation with HyperProp + demon.

Any suggestions how to track reason of such behavor?

[JRC1995]

Hmm...not sure.

[saurabh-kataria]

I had similar experience in my application: speech enhancement. All optimizers gave much worse performance except DemonRanger, which matched performance with my default optimizers (Adam, RAdam, Ranger, AdamW). Although I did not play around with parameters of optimizers much.
I did not experience NaNs in HyperProp BTW. Also, IWA (Iterative Weight Averaging) did not make any difference. I'm wondering if it is similar to SWA (Stochastic Weight Averaging), which helps me BTW.

[JRC1995]

@saurabh-kataria @hadaev8 also when you say all optimizers gave much worse performance, did you mean even the recovered AdamW optimizer (from Readme) from this repo is giving worse performance than, say, your default AdamW?

[hadaev8]

For now, I only tried all optimizers with 2-3 features disabled.
I will try to do it in reverse, start from adamw and add features.
Also, your adamw implementation is not the same as vanilla pytorch, is it intended?

[JRC1995]

@hadaev8 depends on what you mean "not same". On the surface, my implementation is a more general optimizer from which AdamW can be recovered with specific hyperparameters. So from that perspective, it should be of course different from vanilla pytorch. However, if you notice some significant difference in the core AdamW-related operations within the source code let me know. There shouldn't be any intended difference in the operations in the semantic level once the same hyperparameters are used. Though, I don't remember if I ever referred to the vanilla pytorch AdamW. I think I looked at some other repo for reference.

[hadaev8]

You use add while vanilla use mul.
https://github.com/pytorch/pytorch/blob/7403545518a2da056092f401a3e52532280562a4/torch/optim/adamw.py#L73

[JRC1995]

@hadaev8
It should be the same. We need weight - lr*weight_decay*weight.
Both p.mul_(1 - group['lr'] * group['weight_decay']) (vanilla python i.e weight*(1-lr*weight_decay)) and p.add_(group['lr'] * group['weight_decay'], p ) (what I am doing i.e weight - lr*weight_decay*weight) should result in the same thing.
You can verify it using:

x = T.tensor(10.0)
print(x.add_(-0.1, x))
x = T.tensor(10.0)
print(x.mul_(1-0.1))

[saurabh-kataria]

I meant I compared with pytorch internal implementations of Adam, AdamW, etc. For Ranger, I meant comparison with: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
I didn't experiment with "recovered versions of Adam etc."

[hadaev8]

@JRC1995
Yes, you are right, still, I prefer how vanilla pytorch looks like.
I'm running tests now.
Adam extracted from demon ranger seems to be on pair with vanilla.
QHMomentum + AdaMod case nan on validation.

Validation loss 52480:       nan  
Traceback (most recent call last):
  File "train.py", line 487, in <module>
    args.n_gpus, args.local_rank, args.group_name, hparams)
  File "train.py", line 443, in train
    hparams.distributed_run, rank, criterion)
  File "train.py", line 164, in validate
    logger.log_model(model, iteration)
  File "/content/tacotron2/utils/logger.py", line 15, in log_model
    self.add_histogram(tag, value.data.cpu().numpy(), iteration)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/writer.py", line 425, in add_histogram
    histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/summary.py", line 234, in histogram
    hist = make_histogram(values.astype(float), bins, max_bins)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/summary.py", line 272, in make_histogram
    raise ValueError('The histogram is empty, please file a bug report.')
ValueError: The histogram is empty, please file a bug report.

Also
rectify + AdaMod + weight_decay=1e-2 seems to be fine
Also just QHMomentum is fine too.
So its strange.

[JRC1995]

@hadaev8
Yes, I think QHMomentum should be fine if AdamW is fine, because it's mostly a simply extension.
it may be also simply possible that you get "nan" simply because you become unlucky with the random seed, and perhaps the types of models you are running themselves have more tendency towards instability which is for some reason excarbated by certain synergistic combinations and hyperparemeters.

I have actually used AdaMod and QHMomentum together in a project without such issues. The code was a bit different (it was based on the codes here so shouldn't be really different):

from torch.optim.optimizer import Optimizer, required
import torch
import math
import numpy as np


class generic_optimizer(Optimizer):
    def __init__(self, params, config,
                 lr=1e-3, betas=(0.9, 0.999, 0.999), nus=(1.0, 1.0), eps=1e-8,
                 padam_p=0.5, amsgrad=False, AdaMod=False,
                 warmup=False, warmup_steps=2000, use_demon=False,
                 use_gc=False, weight_decay=0.01,
                 lookahead_k=0, lookahead_alpha=0.8,
                 iterate_average=False, iterate_average_cycle=None,
                 epochs=None, steps_per_epoch=None):

        attributes_str = ["lr", "betas", "nus", "padam_p", "eps", "lookahead_k", "lookahead_alpha", "amsgrad", "AdaMod",
                          "use_demon", "warmup", "warmup_steps", "iterate_average", "use_gc", "iterate_average_cycle",
                          "weight_decay"]

        attributes = {}

        for key in attributes_str:
            try:
                attributes[key] = eval("config."+key)
            except:
                attributes[key] = eval(key)

        if not 0.0 <= lr:
            raise ValueError("Invalid learning rate: {}".format(attributes['lr']))
        if not 0.0 <= eps:
            raise ValueError("Invalid epsilon value: {}".format(attributes['eps']))
        if not 0.0 <= padam_p <= 0.5:
            raise ValueError("Invalid padam_p value: {}".format(attributes['padam_p']))
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError("Invalid beta parameter at index 0: {}".format(attributes['betas'][0]))
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError("Invalid beta parameter at index 1: {}".format(attributes['betas'][1]))
        if not 0.0 <= betas[2] < 1.0:
            raise ValueError("Invalid beta parameter at index 2: {}".format(attributes['betas'][2]))
        if not 0.0 <= nus[0] <= 1.0:
            raise ValueError("Invalid nu parameter at index 0: {}".format(attributes['nus'][0]))
        if not 0.0 <= nus[1] <= 1.0:
            raise ValueError("Invalid nu parameter at index 1: {}".format(attributes['nus'][1]))
        if not 0.0 <= lookahead_alpha <= 1.0:
            raise ValueError("Invalid lookahead alpha parameter: {}".format(
                attributes['lookahead_alpha']))
        if attributes['use_demon'] is True and (epochs is None or steps_per_epoch is None):
            raise ValueError(
                "Missing epochs and steps_per_epoch values (Needed if use_demon is True)")

        self.amsgrad = attributes['amsgrad']
        self.AdaMod = attributes['AdaMod']

        self.warmup = attributes['warmup']
        self.warmup_steps = attributes['warmup_steps']
        self.use_demon = attributes['use_demon']

        self.use_gc = attributes['use_gc']

        self.lookahead_k = attributes['lookahead_k']

        self.iterate_average = attributes['iterate_average']
        self.iterate_average_cycle = attributes['iterate_average_cycle']

        self.epochs = epochs
        self.steps_per_epoch = steps_per_epoch
        self.iterate_average_cycle = steps_per_epoch if iterate_average_cycle is None else iterate_average_cycle

        self.T = self.epochs*self.steps_per_epoch

        defaults = dict(lr=attributes['lr'],
                        betas=attributes['betas'],
                        nus=attributes['nus'],
                        eps=attributes['eps'],
                        padam_p=attributes['padam_p'],
                        lookahead_alpha=attributes['lookahead_alpha'],
                        weight_decay=attributes['weight_decay'])
        super(generic_optimizer, self).__init__(params, defaults)

    def __setstate__(self, state):
        super(generic_optimizer, self).__setstate__(state)

    def apply_demon(self, beta1_init, state):
        if self.use_demon:
            temp = 1-(state['step']/self.T)
            beta1 = beta1_init * temp / ((1-beta1_init)+beta1_init*temp)
        else:
            beta1 = beta1_init
        return beta1

    def apply_warmup(self, steps, lr):
        if self.warmup:
            w = min([1.0, steps/self.warmup_steps])
            return w*lr
        return lr

    def decide(self, activate_iterate_average, step):
        iterate_average_step = False
        lookahead_step = False

        if self.iterate_average and activate_iterate_average:
            lookahead_step = False
            if step % self.iterate_average_cycle == 0:
                iterate_average_step = True
        elif self.lookahead_k == 0:
            lookahead_step = False
        else:
            if step % self.lookahead_k == 0:
                lookahead_step = True
            else:
                lookahead_step = False

        return lookahead_step, iterate_average_step

    def step(self, activate_iterate_average=False, closure=None):

        loss = None
        if closure is not None:
            loss = closure()

        for group in self.param_groups:

            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad.data.float()
                if grad.is_sparse:
                    raise RuntimeError('AdvRanger does not support sparse gradients')

                state = self.state[p]

                if len(state) == 0:
                    state['step'] = 0
                    state['exp_avg'] = torch.zeros_like(p.data)
                    state['exp_avg_sq'] = torch.zeros_like(p.data)

                    if self.lookahead_k > 0:
                        state['cached_params'] = p.data.clone()
                    if self.iterate_average:
                        state['num_models'] = 0

                    if self.amsgrad:
                        state['max_exp_avg_sq'] = torch.zeros_like(p.data)
                    if self.AdaMod:
                        state['n_avg'] = torch.zeros_like(p.data)

                if 'cached_params' not in state and self.iterate_average and activate_iterate_average:
                    state['cached_params'] = p.data.clone()

                state['step'] += 1

                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']

                beta1_init, beta2, beta3 = group['betas']
                nu1, nu2 = group['nus']

                beta1 = self.apply_demon(beta1_init, state)
                lr = self.apply_warmup(state['step'], group['lr'])

                wd = group['weight_decay']
                alpha = group['lookahead_alpha']

                lookahead_step, iterate_average_step = self.decide(activate_iterate_average,
                                                                   state['step'])

                if self.use_gc and grad.view(-1).size(0) > 1:
                    grad.add_(-grad.mean(dim=tuple(range(1, len(list(grad.size())))), keepdim=True))

                exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
                exp_avg.mul_(beta1).add_(1 - beta1, grad)

                momentum = exp_avg.clone()
                momentum.div_(1 - (beta1 ** state['step'])).mul_(nu1).add_(1-nu1, grad)

                if wd != 0:
                    p.data.add_(-wd*lr, p.data)

                beta2_t = beta2 ** state['step']

                if self.amsgrad and state['step'] > 1:
                    max_exp_avg_sq = state['max_exp_avg_sq']
                    # Maintains the maximum of all 2nd moment running avg. till now
                    torch.max(max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq)
                    vt = max_exp_avg_sq.clone()
                else:
                    vt = exp_avg_sq.clone()

                bias_correction2 = 1 - beta2_t
                vt.div_(bias_correction2)
                if nu2 != 1.0:
                    vt.mul_(nu2).addcmul_(1-nu2, grad, grad)
                denom = vt.pow_(group['padam_p']).add_(group['eps'])
                n = lr/denom

                if self.AdaMod:
                    n_avg = state['n_avg']
                    n_avg.mul_(beta3).add_(1 - beta3, n)
                    torch.min(n, n_avg, out=n)

                p.data.add_(-n*momentum)

                if lookahead_step:
                    p.data.mul_(alpha).add_(1.0 - alpha, state['cached_params'])
                    state['cached_params'].copy_(p.data)

                if iterate_average_step:
                    p.data.add_(state["num_models"], state['cached_params']
                                ).div_(state["num_models"]+1.0)
                    state['cached_params'].copy_(p.data)
                    state["num_models"] += 1

        return loss

with config:

        self.weight_decay = 1e-4
        self.lr = 1e-3
        self.lookahead_k = 5
        self.use_gc = False  # gradient centralization
        self.AdaMod = True
        self.warmup = True
        self.betas = (0.999, 0.999, 0.999)  # (beta1, beta2, beta3) # beta3 correspond to AdaMod
        self.nus = (0.7, 1.0)  # nu1 and nu2 for Quasi-Hyperbolic momentum
        self.demon = False  # Decaying Momentum

and everything else being the default
and max_grad_norm = 5

grad norm is applied outside the optimizer as:

T.nn.utils.clip_grad_norm_(self.parameters, self.config["max_grad_norm"])

Though in this case grad_norm clipping may be the "magic ingredient" (not sure if I would have or did face any issues without it).

It was a Transformer model with a few differences than the original, the most significant one (in terms of changing the training dynamics) was probably the use of ReZero.