data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.
Code (12)
Tasks
image-classificationImage ClassificationLinguistic AcceptabilityNatural Language InferenceNatural Language UnderstandingParaphrase IdentificationSelf-Supervised LearningSpeech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets
In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extr…
Automatic Speech RecognitionMulti-Task LearningRepresentation LearningSelf-Supervised Learning+2Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the…
Language ModelingLanguage ModellingSpoken Language UnderstandingCocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech
Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic/linguistic domai…
Self-Supervised LearningUnsupervised Data Selection via Discrete Speech Representation for ASR
Self-supervised learning of speech representations has achieved impressive results in improving automatic speech recognition (ASR). In this paper, we show that data selection is important for self-supervised learning. We…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Self-Supervised Learningspeech-recognition+1General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework
This paper presents a self-supervised learning framework, named MGF, for general-purpose speech representation learning. In the design of MGF, speech hierarchy is taken into consideration. Specifically, we propose to use…
ClassificationEmotion ClassificationGeneral ClassificationLanguage Modeling+6