Fast-Slow Transformer for Visually Grounding Speech
We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-VGS. FaST-VGS is a Transformer-based model for learning the associations between raw speech waveforms and visual images. The model unifies dual-encoder and cross-attention architectures into a single model, reaping the superior retrieval speed of the former along with the accuracy of the latter. FaST-VGS achieves state-of-the-art speech-image retrieval accuracy on benchmark datasets, and its learned representations exhibit strong performance on the ZeroSpeech 2021 phonetic and semantic tasks.
Code (1)
Tasks
Image RetrievalRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ID-VTG: Image-Disambiguated Video Temporal Grounding
Video Temporal Grounding (VTG) faces significant challenges when natural language queries must distinguish between multiple events involving visually similar entities, particularly when relying on fine-grained visual att…
Natural Language QueriesSelf-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling
In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge and SUPERB benchmark. Our submissions are based on the recently proposed FaST-VGS model, which is a Transformer-based model that learns to assoc…
Language ModelingLanguage ModellingMasked Language ModelingRepresentation Learning+1Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers
The impressive performances of pre-trained visually grounded language models have motivated a growing body of research investigating what has been learned during the pre-training. As a lot of these models are based on Tr…
Language ModelingLanguage ModellingVisual GroundingThe Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoders
How does visual information included in training affect language processing in audio- and text-based deep learning models? We explore how such visual grounding affects model-internal representations of words, and find su…
Visual GroundingAttention-Based Keyword Localisation in Speech using Visual Grounding
Visually grounded speech models learn from images paired with spoken captions. By tagging images with soft text labels using a trained visual classifier with a fixed vocabulary, previous work has shown that it is possibl…
Visual Grounding