Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to. We demonstrate that these audio-visual associative localizations emerge from network-internal representations learned as a by-product of training to perform an image-audio retrieval task. Our models operate directly on the image pixels and speech waveform, and do not rely on any conventional supervision in the form of labels, segmentations, or alignments between the modalities during training. We perform analysis using the Places 205 and ADE20k datasets demonstrating that our models implicitly learn semantically-coupled object and word detectors.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSound Prompted Semantic SegmentationSpeech Prompted Semantic SegmentationSimilar Papers 제목 키워드 기반
Learning Word-Like Units from Joint Audio-Visual Analysis
Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example,…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)FormLanguage Acquisition+2Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models
Systems that can find correspondences between multiple modalities, such as between speech and images, have great potential to solve different recognition and data analysis tasks in an unsupervised manner. This work studi…
Cross-Modal RetrievalObject LocalizationRetrievalSemantic RetrievalFrom Semi-supervised to Almost-unsupervised Speech Recognition with Very-low Resource by Jointly Learning Phonetic Structures from Audio and Text Embeddings
Producing a large amount of annotated speech data for training ASR systems remains difficult for more than 95% of languages all over the world which are low-resourced. However, we note human babies start to learn the lan…
speech-recognitionSpeech RecognitionUnsupervised Speech RecognitionConnecting Speech to Words through Images
How can we learn the mapping between written words and their spoken counterparts in the absence of explicit textual supervision? We present a visually grounded method for building a vocabulary of spoken words using only …
Image CaptioningKeyword SpottingVisually grounded learning of keyword prediction from untranscribed speech
During language acquisition, infants have the benefit of visual cues to ground spoken language. Robots similarly have access to audio and visual sensors. Recent work has shown that images and spoken captions can be mappe…
Language AcquisitionTAG