paper-with-me

홈 › Papers

Visual Intention Grounding for Egocentric Assistants

2025-04-18 · Pengzhan Sun, Junbin Xiao, Tze Ho Elden Tse, Yicong Li, Arjun Akula, Angela Yao

Visual grounding associates textual descriptions with objects in an image. Conventional methods target third-person image inputs and named object queries. In applications such as AI assistants, the perspective shifts -- inputs are egocentric, and objects may be referred to implicitly through needs and intentions. To bridge this gap, we introduce EgoIntention, the first dataset for egocentric visual intention grounding. EgoIntention challenges multimodal LLMs to 1) understand and ignore unintended contextual objects and 2) reason about uncommon object functionalities. Benchmark results show that current models misidentify context objects and lack affordance understanding in egocentric views. We also propose Reason-to-Ground (RoG) instruction tuning; it enables hybrid training with normal descriptions and egocentric intentions with a chained intention reasoning and object grounding mechanism. RoG significantly outperforms naive finetuning and hybrid training on EgoIntention, while maintaining or slightly improving naive description grounding. This advancement enables unified visual grounding for egocentric and exocentric visual inputs while handling explicit object queries and implicit human intentions.

📄 PDF Abstract BibTeX arXiv:2504.13621

Code (0)

등록된 구현이 없습니다.

Tasks

ObjectVisual Grounding

Similar Papers 제목 키워드 기반

Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions

2024-02-17 · Wenxuan Wang, Yisi Zhang, Xingjian He, Yichen Yan 외

Visual grounding (VG) aims at locating the foreground entities that match the given natural language expressions. Previous datasets and methods for classic VG task mainly rely on the prior assumption that the given expre…

Visual Grounding

EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

2026-08-26 · Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen arxiv

VLMs are increasingly positioned as daily assistants that perceive first-person environments, follow user dialogue, and decide how to help. Existing egocentric benchmarks mainly evaluate visual understanding in isolation…

EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

2026-05-17 · Zeyu Wang, Chang Liu, Eduardus Tjitrahardja, Yuntao Wang 외 arxiv

Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, which is crucial for enabling seamless AI assistant experiences, remains largely overlooked. In this work, we i…

In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting

2025-09-09 · Taiying Peng, Jiacheng Hua, Miao Liu, Feng Lu arxiv

The emergence of advanced multimodal large language models (MLLMs) has significantly enhanced AI assistants' ability to process complex information across modalities. Recently, egocentric videos, by directly capturing us…

Video Question AnsweringGaze Estimation

User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance

2024-08-04 · Mrinal Verghese, Brian Chen, Hamid Eghbalzadeh, Tushar Nagarajan 외

Our research investigates the capability of modern multimodal reasoning models, powered by Large Language Models (LLMs), to facilitate vision-powered assistants for multi-step daily activities. Such assistants must be ab…

Action AnticipationBenchmarkingMultimodal Reasoning