Variance Suppression: Balanced Training Process in Deep Learning
Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obtained is unbalanced. To address this issue, this paper takes the error variance and error mean both into consideration. The adaptively adjusting approach of two terms trading off is also given in our algorithm. Due to this algorithm can suppress error variance, we named it Variance Suppression Gradient Descent (VSSGD). Experimental results have demonstrated that VSSGD can accelerate the training process, effectively prevent overfitting, improve the networks learning capacity from small samples.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSimilar Papers 제목 키워드 기반
Label-Occurrence-Balanced Mixup for Long-tailed Recognition
Mixup is a popular data augmentation method, with many variants subsequently proposed. These methods mainly create new examples via convex combination of random data pairs and their corresponding one-hot labels. However,…
Data AugmentationLow-Rank Structured Clutter Covariance Matrix Estimation for Airborne STAP Radar
In space-time adaptive processing (STAP) of the airborne radar system, it is very important to realize sparse restoration of the clutter covariance matrix with a small number of samples. In this paper, a clutter suppress…
Novel Balanced Single/Dual-band Bandpass Filters Based on Circular Patch Resonator
A new approach for designing the single-band and dual-band balanced bandpass filters (BPFs) based on the circular patch resonator is proposed in this work. By modifying the output balanced ports of the balanced BPFs to i…
TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction
End-to-end training of multi-agent systems offers significant advantages in improving multi-task performance. However, training such models remains challenging and requires extensive manual design and monitoring. In this…
Multi-Task LearningInput correlations impede suppression of chaos and learning in balanced rate networks
Neural circuits exhibit complex activity patterns, both spontaneously and evoked by external stimuli. Information encoding and learning in neural circuits depend on how well time-varying stimuli can control spontaneous n…