Learning to Recognise Words using Visually Grounded Speech
We investigated word recognition in a Visually Grounded Speech model. The model has been trained on pairs of images and spoken captions to create visually grounded embeddings which can be used for speech to image retrieval and vice versa. We investigate whether such a model can be used to recognise words by embedding isolated words and using them to retrieve images of their visual referents. We investigate the time-course of word recognition using a gating paradigm and perform a statistical analysis to see whether well known word competition effects in human speech processing influence word recognition. Our experiments show that the model is able to recognise words, and the gating paradigm reveals that words can be recognised from partial input as well and that recognition is negatively influenced by word competition from the word initial cohort.
Code (0)
등록된 구현이 없습니다.
Tasks
Image RetrievalRetrievalSimilar Papers 제목 키워드 기반
Modelling word learning and recognition using visually grounded speech
Background: Computational models of speech recognition often assume that the set of target words is already given. This implies that these models do not learn to recognise speech from scratch without prior knowledge and …
Representation Learningspeech-recognitionSpeech RecognitionLanguage learning using Speech to Image retrieval
Humans learn language by interaction with their environment and listening to other humans. It should also be possible for computational models to learn language directly from speech but so far most approaches require tex…
Grounded language learningImage RetrievalRetrievalSentence+3Visually Grounded Speech Models for Low-resource Languages and Cognitive Modelling
This dissertation examines visually grounded speech (VGS) models that learn from unlabelled speech paired with images. It focuses on applications for low-resource languages and understanding human language acquisition. W…
Few-Shot LearningLanguage AcquisitionVisually Grounded Speech Models have a Mutual Exclusivity Bias
When children learn new words, they employ constraints such as the mutual exclusivity (ME) bias: a novel word is mapped to a novel object rather than a familiar one. This bias has been studied computationally, but only i…
Spell my name: keyword boosted speech recognition
Recognition of uncommon words such as names and technical terminology is important to understanding conversations in context. However, the ability to recognise such words remains a challenge in modern automatic speech re…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+2