Table Of Contents
Table Of Contents

mxnet.optimizer.Nadam

class mxnet.optimizer.Nadam(learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08, schedule_decay=0.004, **kwargs)[source]

The Nesterov Adam optimizer.

Much like Adam is essentially RMSprop with momentum, Nadam is Adam RMSprop with Nesterov momentum available at http://cs229.stanford.edu/proj2015/054_report.pdf.

This optimizer accepts the following parameters in addition to those accepted by Optimizer.

Parameters
  • beta1 (float, optional) – Exponential decay rate for the first moment estimates.

  • beta2 (float, optional) – Exponential decay rate for the second moment estimates.

  • epsilon (float, optional) – Small value to avoid division by 0.

  • schedule_decay (float, optional) – Exponential decay rate for the momentum schedule

__init__(learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08, schedule_decay=0.004, **kwargs)[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__([learning_rate, beta1, beta2, …])

Initialize self.

create_optimizer(name, **kwargs)

Instantiates an optimizer with a given name and kwargs.

create_state(index, weight)

Creates auxiliary state for a given weight.

create_state_multi_precision(index, weight)

Creates auxiliary state for a given weight, including FP32 high precision copy if original weight is FP16.

register(klass)

Registers a new optimizer.

set_learning_rate(lr)

Sets a new learning rate of the optimizer.

set_lr_mult(args_lr_mult)

Sets an individual learning rate multiplier for each parameter.

set_lr_scale(args_lrscale)

[DEPRECATED] Sets lr scale.

set_wd_mult(args_wd_mult)

Sets an individual weight decay multiplier for each parameter.

update(index, weight, grad, state)

Updates the given parameter using the corresponding gradient and state.

update_multi_precision(index, weight, grad, …)

Updates the given parameter using the corresponding gradient and state.

Attributes

learning_rate

opt_registry