Table Of Contents
Table Of Contents

SyncBatchNorm

class mxnet.gluon.contrib.nn.SyncBatchNorm(in_channels=0, num_devices=None, momentum=0.9, epsilon=1e-05, center=True, scale=True, use_global_stats=False, beta_initializer='zeros', gamma_initializer='ones', running_mean_initializer='zeros', running_variance_initializer='ones', **kwargs)[source]

Cross-GPU Synchronized Batch normalization (SyncBN)

Standard BN [1] implementation only normalize the data within each device. SyncBN normalizes the input within the whole mini-batch. We follow the sync-onece implmentation described in the paper [2].

Parameters:
  • in_channels (int, default 0) – Number of channels (feature maps) in input data. If not specified, initialization will be deferred to the first time forward is called and in_channels will be inferred from the shape of input data.
  • num_devices (int, default number of visible GPUs) –
  • momentum (float, default 0.9) – Momentum for the moving average.
  • epsilon (float, default 1e-5) – Small float added to variance to avoid dividing by zero.
  • center (bool, default True) – If True, add offset of beta to normalized tensor. If False, beta is ignored.
  • scale (bool, default True) – If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling will be done by the next layer.
  • use_global_stats (bool, default False) – If True, use global moving statistics instead of local batch-norm. This will force change batch-norm into a scale shift operator. If False, use local batch-norm.
  • beta_initializer (str or Initializer, default ‘zeros’) – Initializer for the beta weight.
  • gamma_initializer (str or Initializer, default ‘ones’) – Initializer for the gamma weight.
  • moving_mean_initializer (str or Initializer, default ‘zeros’) – Initializer for the moving mean.
  • moving_variance_initializer (str or Initializer, default ‘ones’) – Initializer for the moving variance.
Inputs:
  • data: input tensor with arbitrary shape.
Outputs:
  • out: output tensor with the same shape as data.
Reference:
[1]Ioffe, Sergey, and Christian Szegedy. “Batch normalization: Accelerating deep network training by reducing internal covariate shift.” ICML 2015
[2]Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. “Context Encoding for Semantic Segmentation.” CVPR 2018
__init__(in_channels=0, num_devices=None, momentum=0.9, epsilon=1e-05, center=True, scale=True, use_global_stats=False, beta_initializer='zeros', gamma_initializer='ones', running_mean_initializer='zeros', running_variance_initializer='ones', **kwargs)[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__([in_channels, num_devices, …]) Initialize self.
apply(fn) Applies fn recursively to every child block as well as self.
cast(dtype) Cast this Block to use another data type.
collect_params([select]) Returns a ParameterDict containing this Block and all of its children’s Parameters(default), also can returns the select ParameterDict which match some given regular expressions.
export(path[, epoch]) Export HybridBlock to json format that can be loaded by SymbolBlock.imports, mxnet.mod.Module or the C++ interface.
forward(x, *args) Defines the forward computation.
hybrid_forward(F, x, gamma, beta, …) Overrides to construct symbolic graph for this Block.
hybridize([active]) Activates or deactivates HybridBlock s recursively.
infer_shape(*args) Infers shape of Parameters from inputs.
infer_type(*args) Infers data type of Parameters from inputs.
initialize([init, ctx, verbose, force_reinit]) Initializes Parameter s of this Block and its children.
load_parameters(filename[, ctx, …]) Load parameters from file previously saved by save_parameters.
load_params(filename[, ctx, allow_missing, …]) [Deprecated] Please use load_parameters.
name_scope() Returns a name space object managing a child Block and parameter names.
register_child(block[, name]) Registers block as a child of self.
register_forward_hook(hook) Registers a forward hook on the block.
register_forward_pre_hook(hook) Registers a forward pre-hook on the block.
save_parameters(filename) Save parameters to file.
save_params(filename) [Deprecated] Please use save_parameters.
summary(*inputs) Print the summary of the model’s output and parameters.

Attributes

name Name of this Block, without ‘_’ in the end.
params Returns this Block’s parameter dictionary (does not include its children’s parameters).
prefix Prefix of this Block.