Table Of Contents
Table Of Contents

Perplexity

class mxnet.metric.Perplexity(ignore_label, axis=-1, name='perplexity', output_names=None, label_names=None)[source]

Computes perplexity.

Perplexity is a measurement of how well a probability distribution or model predicts a sample. A low perplexity indicates the model is good at predicting the sample.

The perplexity of a model q is defined as

\[b^{\big(-\frac{1}{N} \sum_{i=1}^N \log_b q(x_i) \big)} = \exp \big(-\frac{1}{N} \sum_{i=1}^N \log q(x_i)\big)\]

where we let b = e.

\(q(x_i)\) is the predicted value of its ground truth label on sample \(x_i\).

For example, we have three samples \(x_1, x_2, x_3\) and their labels are \([0, 1, 1]\). Suppose our model predicts \(q(x_1) = p(y_1 = 0 | x_1) = 0.3\) and \(q(x_2) = 1.0\), \(q(x_3) = 0.6\). The perplexity of model q is \(exp\big(-(\log 0.3 + \log 1.0 + \log 0.6) / 3\big) = 1.77109762852\).

Parameters:
  • ignore_label (int or None) – Index of invalid label to ignore when counting. By default, sets to -1. If set to None, it will include all entries.
  • axis (int (default -1)) – The axis from prediction that was used to compute softmax. By default use the last axis.
  • name (str) – Name of this metric instance for display.
  • output_names (list of str, or None) – Name of predictions that should be used when updating with update_dict. By default include all predictions.
  • label_names (list of str, or None) – Name of labels that should be used when updating with update_dict. By default include all labels.

Examples

>>> predicts = [mx.nd.array([[0.3, 0.7], [0, 1.], [0.4, 0.6]])]
>>> labels   = [mx.nd.array([0, 1, 1])]
>>> perp = mx.metric.Perplexity(ignore_label=None)
>>> perp.update(labels, predicts)
>>> print perp.get()
('Perplexity', 1.7710976285155853)
__init__(ignore_label, axis=-1, name='perplexity', output_names=None, label_names=None)[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__(ignore_label[, axis, name, …]) Initialize self.
get() Returns the current evaluation result.
get_config() Save configurations of metric.
get_name_value() Returns zipped name and value pairs.
reset() Resets the internal evaluation result to initial state.
update(labels, preds) Updates the internal evaluation result.
update_dict(label, pred) Update the internal evaluation with named label and pred