Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
In Multimodal Neural Machine Translation (MNMT), a neural model generates a translated sentence that describes an image, given the image itself and one source descriptions in English. This is considered as the multimodal image caption translation task. The images are processed with Convolutional Neural Network (CNN) to extract visual features exploitable by the translation model. So far, the CNNs used are pre-trained on object detection and localization task. We hypothesize that richer architecture, such as dense captioning models, may be more suitable for MNMT and could lead to improved translations. We extend this intuition to the word-embeddings, where we compute both linguistic and visual representation for our corpus vocabulary. We combine and compare different confi
Code (0)
등록된 구현이 없습니다.
Tasks
Dense CaptioningMachine Translationobject-detectionObject DetectionSentenceTranslationWord EmbeddingsSimilar Papers 제목 키워드 기반
Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge
Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are p…
Grounded language learningImage RetrievalLearning Semantic RepresentationsVisual Grounding+2Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes
We propose a model to learn visually grounded word embeddings (vis-w2v) to capture visual notions of semantic relatedness. While word embeddings trained using text have been extremely successful, they cannot uncover noti…
Common Sense ReasoningImage RetrievalRetrievalVisual Grounding+1How direct is the link between words and images?
Current word embedding models despite their success, still suffer from their lack of grounding in the real world. In this line of research, Gunther et al. 2022 proposed a behavioral experiment to investigate the relation…
Visual GroundingWord EmbeddingsVGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning
Human-annotated attributes serve as powerful semantic embeddings in zero-shot learning. However, their annotation process is labor-intensive and needs expert supervision. Current unsupervised semantic embeddings, i.e., w…
Transfer LearningWord EmbeddingsZero-Shot LearningLearning to Recognise Words using Visually Grounded Speech
We investigated word recognition in a Visually Grounded Speech model. The model has been trained on pairs of images and spoken captions to create visually grounded embeddings which can be used for speech to image retriev…
Image RetrievalRetrieval