ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/
Code (3)
Tasks
3D geometry3D Object Captioning3D Point Cloud Classification3D Point Cloud Linear Classification3D Question Answering (3D-QA)Few-Shot 3D Point Cloud ClassificationGenerative 3D Object ClassificationInstruction FollowingLanguage ModelingLanguage ModellingLarge Language ModelMM-VetMultimodal Large Language ModelObjectVisual GroundingZero-shot 3D classificationZero-Shot Transfer 3D Point Cloud ClassificationSimilar Papers 제목 키워드 기반
ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet b…
3D GenerationLarge Language ModelUniversal Visuo-Tactile Video Understanding for Embodied Interaction
Tactile perception is essential for embodied agents to understand physical attributes of objects that cannot be determined through visual inspection alone. While existing approaches have made progress in visual and langu…
FrictionLarge Language ModelText GenerationVideo UnderstandingLearning Flexible Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for c…
Decision MakingRepresentation LearningLearning Generalizable Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for c…
Decision MakingReinforcement LearningRepresentation LearningYouRefIt: Embodied Reference Understanding with Language and Gesture
We study the understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this new visual task requires understanding mul…