Grounding Multimodal Large Language Models in Actions
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions, we show that a learned tokenization allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action space adapters on five different environments, encompassing over 114 embodied tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
World KnowledgeSimilar Papers 제목 키워드 기반
Multi-Attribute Interactions Matter for 3D Visual Grounding
3D visual grounding aims to localize 3D objects described by free-form language sentences. Following the detection-then-matching paradigm existing methods mainly focus on embedding object attributes in unimodal featu…
3D visual groundingAttributeVisual GroundingTemporal Grounding of Activities using Multimodal Large Language Models
Temporal grounding of activities, the identification of specific time intervals of actions within a larger event context, is a critical task in video understanding. Recent advancements in multimodal large language models…
Video UnderstandingMultimodal Unified Attention Networks for Vision-and-Language Interactions
Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approa…
Question AnsweringVisual GroundingVisual Question AnsweringVisual Question Answering (VQA)MUG: Interactive Multimodal Grounding on User Interfaces
We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen. Prior works modeled multimodal UI grounding in one round: the user gives a command …
Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision
Traditional Visual Grounding (VG) predominantly relies on textual descriptions to localize objects, a paradigm that inherently struggles with linguistic ambiguity and often ignores non-verbal deictic cues prevalent in re…
Referring ExpressionVisual Grounding