Designing a Linearized Potential Function in Neural Network Optimization Using Csiszár Type of Tsallis Entropy
In recent years, learning for neural networks can be viewed as optimization in the space of probability measures. To obtain the exponential convergence to the optimizer, the regularizing term based on Shannon entropy plays an important role. Even though an entropy function heavily affects convergence results, there is almost no result on its generalization, because of the following two technical difficulties: one is the lack of sufficient condition for generalized logarithmic Sobolev inequality, and the other is the distributional dependence of the potential function within the gradient flow equation. In this paper, we establish a framework that utilizes a linearized potential function via Csisz\'{a}r type of Tsallis entropy, which is one of the generalized entropies. We also show that our new framework enable us to derive an exponential convergence result.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization
The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. It is theoretically compelling since it can be seen as a Gaussian process posterior with t…
Bayesian OptimizationDecision MakingGaussian Processesimage-classification+2Cramér-Rao Lower Bounds Arising from Generalized Csiszár Divergences
We study the geometry of probability distributions with respect to a generalized family of Csisz\'ar $f$-divergences. A member of this family is the relative $\alpha$-entropy which is also a R\'enyi analog of relative en…
Linearized Relative Positional Encoding
Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a line…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+2Controllable Neural Architectures for Multi-Task Control
This paper studies a multi-task control problem where multiple linear systems are to be regulated by a single non-linear controller. In particular, motivated by recent advances in multi-task learning and the design of br…
Multi-Task LearningDifferentiable Linearized ADMM
Recently, a number of learning-based optimization methods that combine data-driven architectures with the classical optimization algorithms have been proposed and explored, showing superior empirical performance in solvi…