Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis
The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon is actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then the phenomenon has been thought as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.
Code (1)
Similar Papers 제목 키워드 기반
Analyzing the barren plateau phenomenon in training quantum neural networks with the ZX-calculus
In this paper, we propose a general scheme to analyze the gradient vanishing phenomenon, also known as the barren plateau phenomenon, in training quantum neural networks with the ZX-calculus. More precisely, we extend th…
The Origin of Quantum Mechanical Statistics: Some Insights from the Research on Human Language
Identical systems, or entities, are indistinguishable in quantum mechanics (QM), and the symmetrization postulate rules the possible statistical distributions of a large number of identical quantum entities. However, a t…
Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance
The ability of neural networks to provide `best in class' approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable …
On barren plateaus and cost function locality in variational quantum algorithms
Variational quantum algorithms rely on gradient based optimization to iteratively minimize a cost function evaluated by measuring output(s) of a quantum processor. A barren plateau is the phenomenon of exponentially vani…
On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages
The multi-stage phenomenon in the training loss curves of neural networks has been widely observed, reflecting the non-linearity and complexity inherent in the training process. In this work, we investigate the training …