Self-supervised Document Clustering Based on BERT with Data Augment
Contrastive learning is a promising approach to unsupervised learning, as it inherits the advantages of well-studied deep models without a dedicated and complex model design. In this paper, based on bidirectional encoder representations from transformers, we propose self-supervised contrastive learning (SCL) as well as few-shot contrastive learning (FCL) with unsupervised data augmentation (UDA) for text clustering. SCL outperforms state-of-the-art unsupervised clustering approaches for short texts and those for long texts in terms of several clustering evaluation measures. FCL achieves performance close to supervised learning, and FCL with UDA further improves the performance for short texts.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringContrastive LearningData AugmentationText ClusteringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pac-HuBERT: Self-Supervised Music Source Separation via Primitive Auditory Clustering and Hidden-Unit BERT
In spite of the progress in music source separation research, the small amount of publicly-available clean source data remains a constant limiting factor for performance. Thus, recent advances in self-supervised learning…
ClusteringDecoderMusic Source SeparationSelf-Supervised LearningUnsupervised Contextualized Document Representation
Several NLP tasks need the effective representation of text documents. Arora et. al., 2017 demonstrate that simple weighted averaging of word vectors frequently outperforms neural models. SCDV (Mekala et. al., 2017) furt…
ClusteringSentenceSentence SimilarityWord Sense DisambiguationShuffle & Divide: Contrastive Learning for Long Text
We propose a self-supervised learning method for long text documents based on contrastive learning. A key to our method is Shuffle and Divide (SaD), a simple text augmentation algorithm that sets up a pretext task requir…
Contrastive LearningDocument EmbeddingSelf-Supervised LearningText Augmentation+3HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the p…
ClusteringLanguage ModellingRepresentation LearningSpeech Recognition+1Subtopic Clustering with a Query-Specific Siamese Similarity Metric
We propose a Query-Specific Siamese Similarity Metric (QS3M) for query-specific clustering of text documents. It uses fine-tuned BERT embeddings and trains a non-linear projection into a query-specific similarity space. …
Clustering