CERT: Contrastive Self-supervised Learning for Language Understanding
Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingSelf-Supervised LearningSentenceTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SSL-SLR: Self-Supervised Representation Learning for Sign Language Recognition
Sign language recognition (SLR) is a machine learning task aiming to identify signs in videos. Due to the scarcity of annotated data, unsupervised methods like contrastive learning have become promising in this field. Th…
Sign Language RecognitionSelf-Supervised LearningRepresentation LearningContrastive LearningLabel Anchored Contrastive Learning for Language Understanding
Contrastive learning (CL) has achieved astonishing progress in computer vision, speech, and natural language processing fields recently with self-supervised learning. However, CL approach to the supervised setting is not…
BenchmarkingContrastive LearningData AugmentationNatural Language Understanding+1A Theoretical Study of Inductive Biases in Contrastive Learning
Understanding self-supervised learning is important but challenging. Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes. However, the recent work of Saunshi e…
ClusteringContrastive LearningSelf-Supervised LearningOn the duality between contrastive and non-contrastive self-supervised learning
Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While diffe…
Self-Supervised LearningUnderstanding Self-supervised Contrastive Learning through Supervised Objectives
Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised repr…
Self-Supervised LearningRepresentation LearningContrastive Learning