Score-Based Training for Energy-Based TTS Models
Noise contrastive estimation (NCE) is a popular method for training energy-based models (EBM) with intractable normalisation terms. The key idea of NCE is to learn by comparing unnormalised log-likelihoods of the reference and noisy samples, thus avoiding explicitly computing normalisation terms. However, NCE critically relies on the quality of noisy samples. Recently, sliced score matching (SSM) has been popularised by closely related diffusion models (DM). Unlike NCE, SSM learns a gradient of log-likelihood, or score, by learning distribution of its projections on randomly chosen directions. However, both NCE and SSM disregard the form of log-likelihood function, which is problematic given that EBMs and DMs make use of first-order optimisation during inference. This paper proposes a new criterion that learns scores more suitable for first-order schemes. Experiments contrasts these approaches for training EBMs.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Energy-based Out-of-distribution Detection
Determining whether inputs are out-of-distribution (OOD) is an essential building block for safely deploying machine learning models in the open world. However, previous methods relying on the softmax confidence score su…
Out-of-Distribution DetectionEnergy Discrepancies: A Score-Independent Loss for Energy-Based Models
Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Dis…
DecoderShould EBMs model the energy or the score?
Recent progress in training unnormalized models through denoising score matching with Langevin dynamics (SMLD) and denoising diffusion probabilistic modeling (DDPM) has made unnormalized models a competitive model class …
DenoisingmodelEnergy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
The discrepancy between maximum likelihood estimation (MLE) and task measures such as BLEU score has been studied before for autoregressive neural machine translation (NMT) and resulted in alternative training algorithms…
Computational EfficiencyMachine TranslationNMTReranking+3Image Quality Assessment Techniques Improve Training and Evaluation of Energy-Based Generative Adversarial Networks
We propose a new, multi-component energy function for energy-based Generative Adversarial Networks (GANs) based on methods from the image quality assessment literature. Our approach expands on the Boundary Equilibrium Ge…
Generative Adversarial NetworkImage Quality Assessment