Text Transformations in Contrastive Self-Supervised Learning: A Review
Contrastive self-supervised learning has become a prominent technique in representation learning. The main step in these methods is to contrast semantically similar and dissimilar pairs of samples. However, in the domain of Natural Language Processing (NLP), the augmentation methods used in creating similar pairs with regard to contrastive learning (CL) assumptions are challenging. This is because, even simply modifying a word in the input might change the semantic meaning of the sentence, and hence, would violate the distributional hypothesis. In this review paper, we formalize the contrastive learning framework, emphasize the considerations that need to be addressed in the data transformation step, and review the state-of-the-art methods and evaluations for contrastive representation learning in NLP. Finally, we describe some challenges and potential directions for learning better text representations using contrastive methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningRepresentation LearningSelf-Supervised LearningSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Supervised Learning for Group Equivariant Neural Networks
This paper proposes a method to construct pretext tasks for self-supervised learning on group equivariant neural networks. Group equivariant neural networks are the models whose structure is restricted to commute with th…
Self-Supervised LearningA brief review of contrastive learning applied to astrophysics
Reliable tools to extract patterns from high-dimensionality spaces are becoming more necessary as astronomical datasets increase both in volume and complexity. Contrastive Learning is a self-supervised machine learning a…
AstronomyContrastive LearningOn Compositions of Transformations in Contrastive Self-Supervised Learning
In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning. In this paper, we generalize contrastive learning to a wider set …
Action RecognitionAudio ClassificationContrastive LearningRepresentation Learning+1Can Temporal Information Help with Contrastive Self-Supervised Learning?
Leveraging temporal information has been regarded as essential for developing video understanding models. However, how to properly incorporate temporal information into the recent successful instance discrimination based…
Data AugmentationRepresentation LearningSelf-Supervised LearningVideo UnderstandingThe Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning
Contrastive learning of auditory and visual perception has been extremely successful when investigated individually. However, there are still major questions on how we could integrate principles learned from both domains…
Contrastive LearningRepresentation LearningSelf-Supervised Learning