AttentionMix: Data augmentation method that relies on BERT attention mechanism
The Mixup method has proven to be a powerful data augmentation technique in Computer Vision, with many successors that perform image mixing in a guided manner. One of the interesting research directions is transferring the underlying Mixup idea to other domains, e.g. Natural Language Processing (NLP). Even though there already exist several methods that apply Mixup to textual data, there is still room for new, improved approaches. In this work, we introduce AttentionMix, a novel mixing method that relies on attention-based information. While the paper focuses on the BERT attention mechanism, the proposed approach can be applied to generally any attention-based model. AttentionMix is evaluated on 3 standard sentiment classification datasets and in all three cases outperforms two benchmark approaches that utilize Mixup mechanism, as well as the vanilla BERT method. The results confirm that the attention-based information can be effectively used for data augmentation in the NLP domain.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationSentiment AnalysisSentiment ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data
We propose AttentionMixer, a unified deep learning framework for multimodal detection of brain edema that combines structural head CT (HCT) with routine clinical metadata. While HCT provides rich spatial information, cli…
Multimodal Deep LearningAttentionMixer: An Accurate and Interpretable Framework for Process Monitoring
An accurate and explainable automatic monitoring system is critical for the safety of high efficiency energy conversion plants that operate under extreme working condition. Nonetheless, currently available data-driven mo…
Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
Contrastive learning has been used to learn a high-quality representation of the image in computer vision. However, contrastive learning is not widely utilized in natural language processing due to the lack of a general …
Contrastive LearningData AugmentationRelationRelation ExtractionThe Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data
Recently, backdoor attacks have posed a serious security threat to the training process of deep neural networks (DNNs). The attacked model behaves normally on benign samples but outputs a specific result when the trigger…
Data AugmentationmodelCross-Probe BERT for Efficient and Effective Cross-Modal Search
Inspired by the great success of BERT in NLP tasks, many text-vision BERT models emerged recently. Benefited from cross-modal attention, text-vision BERT models have achieved excellent performance in many language-visio…
Image RetrievalRetrieval