Bounds all around: training energy-based models with bidirectional bounds
Energy-based models (EBMs) provide an elegant framework for density estimation, but they are notoriously difficult to train. Recent work has established links to generative adversarial networks, where the EBM is trained through a minimax game with a variational value function. We propose a bidirectional bound on the EBM log-likelihood, such that we maximize a lower bound and minimize an upper bound when solving the minimax game. We link one bound to a gradient penalty that stabilizes training, thereby providing grounding for best engineering practice. To evaluate the bounds we develop a new and efficient estimator of the Jacobi-determinant of the EBM generator. We demonstrate that these developments significantly stabilize training and yield high-quality density estimation and sample generation.
Code (0)
등록된 구현이 없습니다.
Tasks
AllDensity EstimationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring bidirectional bounds for minimax-training of Energy-based models
Energy-based models (EBMs) estimate unnormalized densities in an elegant framework, but they are generally difficult to train. Recent work has linked EBMs to generative adversarial networks, by noting that they can be tr…
Density EstimationOn generalization bounds for deep networks based on loss surface implicit regularization
The classical statistical learning theory implies that fitting too many parameters leads to overfitting and poor performance. That modern deep neural networks generalize well despite a large number of parameters contradi…
Generalization BoundsLearning TheoryTrajectory-Independent Flexibility Envelopes of Energy-Constrained Systems with State-Dependent Losses
As non-dispatchable renewable power units become prominent in electric power grids, demand-side flexibility appears as a key element of future power systems' operation. Power and energy bounds are intuitive metrics to de…
Energy-efficiency Limits on Training AI Systems using Learning-in-Memory
Learning-in-memory (LIM) is a recently proposed paradigm to overcome fundamental memory bottlenecks in training machine learning systems. While compute-in-memory (CIM) approaches can address the so-called memory-wall (i.…
Decentralized and Privacy-Preserving Learning of Approximate Stackelberg Solutions in Energy Trading Games with Demand Response Aggregators
In this work, a novel Stackelberg game theoretic framework is proposed for trading energy bidirectionally between the demand-response (DR) aggregator and the prosumers. This formulation allows for flexible energy arbitra…
energy tradingPrivacy Preserving