Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes

2015-11-22 · CVPR 2016 6 · Satwik Kottur, Ramakrishna Vedantam, José M. F. Moura, Devi Parikh

We propose a model to learn visually grounded word embeddings (vis-w2v) to capture visual notions of semantic relatedness. While word embeddings trained using text have been extremely successful, they cannot uncover notions of semantic relatedness implicit in our visual world. For instance, although "eats" and "stares at" seem unrelated in text, they share semantics visually. When people are eating something, they also tend to stare at the food. Grounding diverse relations like "eats" and "stares at" into vision remains challenging, despite recent progress in vision. We note that the visual grounding of words depends on semantics, and not the literal pixels. We thus use abstract scenes created from clipart to provide the visual grounding. We find that the embeddings we learn capture fine-grained, visually grounded notions of semantic relatedness. We show improvements over text-only word embeddings (word2vec) on three tasks: common-sense assertion classification, visual paraphrasing and text-based image retrieval. Our code and datasets are available online.

📄 PDF Abstract BibTeX arXiv:1511.07067

Code (1)

satwikkottur/VisualWord2Vec

Tasks

Common Sense Reasoning Image Retrieval Retrieval Visual GroundingWord Embeddings

Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

2022-02-21 · CMCL (ACL) 2022 5 · Danny Merkx, Stefan L. Frank, Mirjam Ernestus

Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are p…

Grounded language learning Image Retrieval Learning Semantic Representations Visual Grounding+2

Learning to Recognise Words using Visually Grounded Speech

2020-05-31 · Sebastiaan Scholten, Danny Merkx, Odette Scharenborg

We investigated word recognition in a Visually Grounded Speech model. The model has been trained on pairs of images and spoken captions to create visually grounded embeddings which can be used for speech to image retriev…

Image Retrieval Retrieval

How direct is the link between words and images?

2022-06-30 · Hassan Shahmohammadi, Maria Heitmeier, Elnaz Shafaei-Bajestan, Hendrik P. A. Lensch 외

Current word embedding models despite their success, still suffer from their lack of grounding in the real world. In this line of research, Gunther et al. 2022 proposed a behavioral experiment to investigate the relation…

Visual Grounding Word Embeddings

VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning

2022-03-20 · CVPR 2022 1 · Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele 외

Human-annotated attributes serve as powerful semantic embeddings in zero-shot learning. However, their annotation process is labor-intensive and needs expert supervision. Current unsupervised semantic embeddings, i.e., w…

Transfer Learning Word Embeddings Zero-Shot Learning

Leveraging Pretrained Image-text Models for Improving Audio-Visual Learning

2023-09-08 · Saurabhchand Bhati, Jesús Villalba, Laureano Moro-Velazquez, Thomas Thebaud 외

Visually grounded speech systems learn from paired images and their spoken captions. Recently, there have been attempts to utilize the visually grounded models trained from images and their corresponding text captions, s…

audio-visual learning Quantization Word Embeddings