SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes
Existing 3D scene understanding methods are heavily focused on 3D semantic and instance segmentation. However identifying objects and their parts only constitutes an intermediate step towards a more fine-grained goal which is effectively interacting with the functional interactive elements (e.g. handles knobs buttons) in the scene to accomplish diverse tasks. To this end we introduce SceneFun3D a large-scale dataset with more than 14.8k highly accurate interaction annotations for 710 high-resolution real-world 3D indoor scenes. We accompany the annotations with motion parameter information describing how to interact with these elements and a diverse set of natural language descriptions of tasks that involve manipulating them in the scene context. To showcase the value of our dataset we introduce three novel tasks namely functionality segmentation task-driven affordance grounding and 3D motion estimation and adapt existing state-of-the-art methods to tackle them. Our experiments show that solving these tasks in real 3D scenes remains challenging despite recent progress in closed-set and open-set 3D scene understanding methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Instance SegmentationMotion EstimationScene UnderstandingSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
Effective human-agent collaboration in physical environments requires understanding not only what to act upon, but also where the actionable elements are and how to interact with them. Existing approaches often operate a…
Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement
Understanding 3D scene-level affordances from natural language instructions is essential for enabling embodied agents to interact meaningfully in complex environments. However, this task remains challenging due to the ne…
Affordance DetectionPoint CloudsFunctionality understanding and segmentation in 3D scenes
Understanding functionalities in 3D scenes involves interpreting natural language descriptions to locate functional interactive objects, such as handles and buttons, in a 3D environment. Functionality understanding is hi…
AI AgentLanguage ModelingLanguage ModellingWorld KnowledgeGrounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
Functional affordance grounding requires more than recognizing an object: an agent must localize the specific region that supports an interaction, such as the handle to pull or the button to press. This is difficult for …
Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videos
Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action antici…
Action AnticipationAction RecognitionAffordance RecognitionHuman-Object Interaction Detection+2