From Random to Supervised: A Novel Dropout Mechanism Integrated with Global Information
Dropout is used to avoid overfitting by randomly dropping units from the neural networks during training. Inspired by dropout, this paper presents GI-Dropout, a novel dropout method integrating with global information to improve neural networks for text classification. Unlike the traditional dropout method in which the units are dropped randomly according to the same probability, we aim to use explicit instructions based on global information of the dataset to guide the training process. With GI-Dropout, the model is supposed to pay more attention to inapparent features or patterns. Experiments demonstrate the effectiveness of the dropout with global information on seven text classification tasks, including sentiment analysis and topic classification.
Code (1)
Tasks
ClassificationGeneral ClassificationSentiment Analysistext-classificationText ClassificationTopic ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dropping Anchor and Spherical Harmonics for Sparse-view Gaussian Splatting
Recent 3D Gaussian Splatting (3DGS) Dropout methods address overfitting under sparse-view conditions by randomly nullifying Gaussian opacities. However, we identify a neighbor compensation effect in these approaches: dro…
Model CompressionSynaptic Pruning: A Biological Inspiration for Deep Learning Regularization
Synaptic pruning in biological brains removes weak connections to improve efficiency. In contrast, dropout regularization in artificial neural networks randomly deactivates neurons without considering activity-dependent …
Time Series ForecastingEmulation-based Stabilization for Networked Control Systems with Stochastic Channels
This paper studies the stabilization problem of networked control systems (NCSs) with random packet dropouts caused by stochastic channels. To describe the effects of stochastic channels on the information transmission, …
SchedulingDropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation
Predicting the altered acoustic frames is an effective way of self-supervised learning for speech representation. However, it is challenging to prevent the pretrained model from overfitting. In this paper, we proposed to…
ClassificationSelf-Supervised LearningSpeaker RecognitionScheduled DropHead: A Regularization Method for Transformer Models
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various N…
Machine Translationtext-classificationText ClassificationTranslation