Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling
In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge and SUPERB benchmark. Our submissions are based on the recently proposed FaST-VGS model, which is a Transformer-based model that learns to associate raw speech waveforms with semantically related images, all without the use of any transcriptions of the speech. Additionally, we introduce a novel extension of this model, FaST-VGS+, which is learned in a multi-task fashion with a masked language modeling objective in addition to the visual grounding objective. On ZeroSpeech 2021, we show that our models perform competitively on the ABX task, outperform all other concurrent submissions on the Syntactic and Semantic tasks, and nearly match the best system on the Lexical task. On the SUPERB benchmark, we show that our models also achieve strong performance, in some cases even outperforming the popular wav2vec2.0 model.
Code (1)
Tasks
Language ModelingLanguage ModellingMasked Language ModelingRepresentation LearningVisual GroundingSimilar Papers 제목 키워드 기반
Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly …
Self-Supervised LearningRepresentation LearningSpeech RecognitionVisual GroundingMapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding
In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speec…
Visual GroundingImage CaptioningKeyword SpottingSyllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model
In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective. We demonstrate that a nearly identical model architect…
Language ModelingLanguage ModellingMasked Language ModelingSegmentation+2Text-Free Image-to-Speech Synthesis Using Learned Segmental Units
In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supe…
Image CaptioningSpeech SynthesisVisual GroundingBridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech
Learning to understand grounded language, which connects natural language to percepts, is a critical research area. Prior work in grounded language acquisition has focused primarily on textual inputs. In this work we dem…
Language Acquisitionspeech-recognitionSpeech Recognition