Regularizing Neural Networks by Penalizing Confident Output Distributions

2017-01-23 · Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, Geoffrey Hinton

We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.

📄 PDF Abstract BibTeX arXiv:1701.06548

Code (2)

hyang0129/foodclassapp tf

makeyourownmaker/mixup pytorch

Tasks

General Classification image-classification Image ClassificationLanguage ModelingLanguage Modelling Machine Translation Reinforcement Learning Reinforcement Learning (RL)speech-recognition Speech Recognition Translation

Methods 이 논문이 사용한 방법론

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Midpoint Regularization: from High Uncertainty Training to Conservative Classification

2021-06-26 · Hongyu Guo

Label Smoothing (LS) improves model generalization through penalizing models from generating overconfident output distributions. For each training sample the LS strategy smooths the one-hot encoded training signal by dis…

Classification Vocal Bursts Intensity Prediction

Unifying Input and Output Smoothing in Neural Machine Translation

2020-12-01 · COLING 2020 8 · Yingbo Gao, Baohao Liao, Hermann Ney

Soft contextualized data augmentation is a recent method that replaces one-hot representation of words with soft posterior distributions of an external language model, smoothing the input of neural machine translation sy…

Data Augmentation Language Modeling Language Modelling Machine Translation+1

Causal Regularization

2019-06-28 · NeurIPS 2019 12 · Dominik Janzing

I argue that regularizing terms in standard regression methods not only help against overfitting finite data, but sometimes also yield better causal models in the infinite sample regime. I first consider a multi-dimensio…

Learning Theory regression

Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing

2020-05-02 · ACL 2020 6 · Clara Meister, Elizabeth Salesky, Ryan Cotterell

Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i.e. over-confident) predictions, a common sign of overfitting. This class of techniques, of which label …

Text Generation

Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation

2019-10-10 · Theodoros Tsiligkaridis

Precise estimation of uncertainty in predictions for AI systems is a critical factor in ensuring trust and safety. Deep neural networks trained with a conventional method are prone to over-confident predictions. In contr…