Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax
The Gumbel-Softmax is a continuous distribution over the simplex that is often used as a relaxation of discrete distributions. Because it can be readily interpreted and easily reparameterized, it enjoys widespread use. We propose a modular and more flexible family of reparameterizable distributions where Gaussian noise is transformed into a one-hot approximation through an invertible function. This invertible function is composed of a modified softmax and can incorporate diverse transformations that serve different specific purposes. For example, the stick-breaking procedure allows us to extend the reparameterization trick to distributions with countably infinite support, thus enabling the use of our distribution along nonparametric models, or normalizing flows let us increase the flexibility of the distribution. Our construction enjoys theoretical advantages over the Gumbel-Softmax, such as closed form KL, and significantly outperforms it in a variety of experiments. Our code is available at https://github.com/cunningham-lab/igr.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Categorical Reparameterization with Gumbel-Softmax
Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through sampl…
General ClassificationJoint Beamforming and Integer User Association using a GNN with Gumbel-Softmax Reparameterizations
Machine learning (ML) models can effectively optimize a multi-cell wireless network by designing the beamforming vectors and association decisions. Existing ML designs, however, often needs to approximate the integer ass…
Graph Neural NetworkGeneralized Gumbel-Softmax Gradient Estimator for Generic Discrete Random Variables
Estimating the gradients of stochastic nodes in stochastic computational graphs is one of the crucial research questions in the deep generative modeling community, which enables the gradient descent optimization on neura…
Topic ModelsNeural Machine Translation with Gumbel-Greedy Decoding
Previous neural machine translation models used some heuristic search algorithms (e.g., beam search) in order to avoid solving the maximum a posteriori problem over translation sentences at test time. In this paper, we p…
Heuristic SearchMachine TranslationTranslationGumbel-Softmax Selective Networks
ML models often operate within the context of a larger system that can adapt its response when the ML model is uncertain, such as falling back on safe defaults or a human in the loop. This commonly encountered operationa…