InViG: Benchmarking Interactive Visual Grounding with 500K Human-Robot Interactions

2023-10-18 · Hanbo Zhang, Jie Xu, Yuchen Mo, Tao Kong

Ambiguity is ubiquitous in human communication. Previous approaches in Human-Robot Interaction (HRI) have often relied on predefined interaction templates, leading to reduced performance in realistic and open-ended scenarios. To address these issues, we present a large-scale dataset, \invig, for interactive visual grounding under language ambiguity. Our dataset comprises over 520K images accompanied by open-ended goal-oriented disambiguation dialogues, encompassing millions of object instances and corresponding question-answer pairs. Leveraging the \invig dataset, we conduct extensive studies and propose a set of baseline solutions for end-to-end interactive visual disambiguation and grounding, achieving a 45.6\% success rate during validation. To the best of our knowledge, the \invig dataset is the first large-scale dataset for resolving open-ended interactive visual grounding, presenting a practical yet highly challenging benchmark for ambiguity-aware HRI. Codes and datasets are available at: \href{https://openivg.github.io}{https://openivg.github.io}.

📄 PDF Abstract BibTeX arXiv:2310.12147

Code (1)

zhanghanbo/invig-dataset

Tasks

Benchmarking Visual Grounding

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

INVIGORATE: Interactive Visual Grounding and Grasping in Clutter

2021-08-25 · Hanbo Zhang, Yunfan Lu, Cunjun Yu, David Hsu 외

This paper presents INVIGORATE, a robot system that interacts with human through natural language and grasps a specified object in clutter. The objects may occlude, obstruct, or even stack on top of one another. INVIGORA…

Blocking Object object-detection Object Detection+3

CLUE: Crossmodal disambiguation via Language-vision Understanding with attEntion

2026-02-09 · Mouad Abrini, Mohamed Chetouani arxiv

With the increasing integration of robots into daily life, human-robot interaction has become more complex and multifaceted. A critical component of this interaction is Interactive Visual Grounding (IVG), through which r…

parameter-efficient fine-tuning Visual Grounding

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

2026-08-25 · Zhengxiang Wang, Owen Rambow arxiv

Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a central property of real-world reference: initial referring expressions…

Referring Expression Visual Grounding

Interactive Reinforcement Learning for Object Grounding via Self-Talking

2017-12-02 · Yan Zhu, Shaoting Zhang, Dimitris Metaxas

Humans are able to identify a referred visual object in a complex scene via a few rounds of natural language communications. Success communication requires both parties to engage and learn to adapt for each other. In thi…

Object reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction

2018-06-11 · Mohit Shridhar, David Hsu

This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their rela…

Question Generation Question-Generation Visual Grounding