Interactive Reinforcement Learning for Object Grounding via Self-Talking

2017-12-02 · Yan Zhu, Shaoting Zhang, Dimitris Metaxas

Humans are able to identify a referred visual object in a complex scene via a few rounds of natural language communications. Success communication requires both parties to engage and learn to adapt for each other. In this paper, we introduce an interactive training method to improve the natural language conversation system for a visual grounding task. During interactive training, both agents are reinforced by the guidance from a common reward function. The parametrized reward function also cooperatively updates itself via interactions, and contribute to accomplishing the task. We evaluate the method on GuessWhat?! visual grounding task, and significantly improve the task success rate. However, we observe language drifting problem during training and propose to use reward engineering to improve the interpretability for the generated conversations. Our result also indicates evaluating goal-ended visual conversation tasks require semantic relevant metrics beyond task success rate.

📄 PDF Abstract BibTeX arXiv:1712.00576

Code (0)

등록된 구현이 없습니다.

Tasks

Objectreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Visual Grounding

Methods 이 논문이 사용한 방법론

Interpretability 설명 없음

Learning a Policy for Opportunistic Active Learning

2018-08-29 · EMNLP 2018 10 · Aishwarya Padmakumar, Peter Stone, Raymond J. Mooney

Active learning identifies data points to label that are expected to be the most useful in improving a supervised model. Opportunistic active learning incorporates active learning into interactive tasks that constrain po…

Active Learning Object reinforcement-learning Reinforcement Learning+2

Conversational Human Audio-visual Talking Dialogue Generation

2026-07-02 · Junhao Song, Lluis Guasch, Xilin He, Zhongyu Yang 외 arxiv

Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consumi…

Dialogue Generation

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

2023-02-06 · Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier 외

Recent works successfully leveraged Large Language Models' (LLM) abilities to capture abstract knowledge about world's physics to solve decision-making problems. Yet, the alignment between LLMs' knowledge and the environ…

Decision Making reinforcement-learning Reinforcement Learning (RL)

Multi-human Interactive Talking Dataset

2025-08-05 · Zeyu Zhu, Weijia Wu, Mike Zheng Shou arxiv

Existing studies on talking video generation have predominantly focused on single-person monologues or isolated facial animations, limiting their applicability to realistic multi-human interactions. To bridge this gap, w…

Video Generation

TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation

2025-12-23 · Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Joon Son Chung 외 arxiv

The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have …