Ruminating Word Representations with Random Noise Masking
We introduce a training method for better word representation and performance, which we call \textbf{GraVeR} (\textbf{Gra}dual \textbf{Ve}ctor \textbf{R}umination). The method is to gradually and iteratively add random noises and bias to word embeddings after training a model, and re-train the model from scratch but initialize with the noised word embeddings. Through the re-training process, some of noises can be compensated and other noises can be utilized to learn better representations. As a result, we can get word representations further fine-tuned and specialized in the task. On six text classification tasks, our method improves model performances with a large gap. When GraVeR is combined with other regularization techniques, it shows further improvements. Lastly, we investigate the usefulness of GraVeR for pretraining by training data.
Code (0)
등록된 구현이 없습니다.
Tasks
text-classificationText ClassificationWord EmbeddingsSimilar Papers 제목 키워드 기반
Ruminating Word Representations with Random Noised Masker
We introduce a training method for both better word representation and performance, which we call GROVER (Gradual Rumination On the Vector with maskERs). The method is to gradually and iteratively add random noises to wo…
text-classificationText ClassificationWord EmbeddingsColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders
Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework, offering remarkable performance across a wide range of downstream tasks. To increase the difficulty of the pretext task and learn richer visua…
Image ClassificationInstance SegmentationObject DetectionSemantic SegmentationDo Blind Spots Matter for Word-Referent Mapping? A Computational Study with Infant Egocentric Video
Typically, children start to learn their first words between 6 and 9 months, linking spoken utterances to their visual referents. Without prior knowledge, a word encountered for the first time can be interpreted in count…
Contrastive LearningPre-training with Random Orthogonal Projection Image Modeling
Masked Image Modeling (MIM) is a powerful self-supervised strategy for visual pre-training without the use of labels. MIM applies random crops to input images, processes them with an encoder, and then recovers the masked…
DecoderJoint Low-level and High-level Textual Representation Learning with Multiple Masking Strategies
Most existing text recognition methods are trained on large-scale synthetic datasets due to the scarcity of labeled real-world datasets. Synthetic images, however, cannot faithfully reproduce real-world scenarios, such a…
Contrastive LearningImage Super-ResolutionRepresentation LearningSelf-Supervised Learning+1