Unsupervised Online Grounding of Natural Language during Human-Robot Interactions

2020-07-05 · WS 2020 7 · Oliver Roesler

Allowing humans to communicate through natural language with robots requires connections between words and percepts. The process of creating these connections is called symbol grounding and has been studied for nearly three decades. Although many studies have been conducted, not many considered grounding of synonyms and the employed algorithms either work only offline or in a supervised manner. In this paper, a cross-situational learning based grounding framework is proposed that allows grounding of words and phrases through corresponding percepts without human supervision and online, i.e. it does not require any explicit training phase, but instead updates the obtained mappings for every new encountered situation. The proposed framework is evaluated through an interaction experiment between a human tutor and a robot, and compared to an existing unsupervised grounding framework. The results show that the proposed framework is able to ground words through their corresponding percepts online and in an unsupervised manner, while outperforming the baseline framework.

📄 PDF Abstract BibTeX arXiv:2007.04304

Code (0)

등록된 구현이 없습니다.

Natural Language Grounding and Grammar Induction for Robotic Manipulation Commands

2017-08-01 · WS 2017 8 · Muhannad Alomari, Paul Duckworth, Majd Hawasly, David C. Hogg 외

We present a cognitively plausible system capable of acquiring knowledge in language and vision from pairs of short video clips and linguistic descriptions. The aim of this work is to teach a robot manipulator how to exe…

Generating Descriptions with Grounded and Co-Referenced People

2017-04-05 · CVPR 2017 7 · Anna Rohrbach, Marcus Rohrbach, Siyu Tang, Seong Joon Oh 외

Learning how to generate descriptions of images or videos received major interest both in the Computer Vision and Natural Language Processing communities. While a few works have proposed to learn a grounding during the g…

Interactive Reinforcement Learning for Object Grounding via Self-Talking

2017-12-02 · Yan Zhu, Shaoting Zhang, Dimitris Metaxas

Humans are able to identify a referred visual object in a complex scene via a few rounds of natural language communications. Success communication requires both parties to engage and learn to adapt for each other. In thi…

Object reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Learning Unsupervised Visual Grounding Through Semantic Self-Supervision

2018-03-17 · Syed Ashar Javed, Shreyas Saxena, Vineet Gandhi

Localizing natural language phrases in images is a challenging problem that requires joint understanding of both the textual and visual modalities. In the unsupervised setting, lack of supervisory signals exacerbate this…

Visual Grounding

A Game Interface to Study Semantic Grounding in Text-Based Models

2021-08-17 · Timothee Mickus, Mathieu Constant, Denis Paperno

Can language models learn grounded representations from text distribution alone? This question is both central and recurrent in natural language processing; authors generally agree that grounding requires more than textu…