Learning to Label Affordances from Simulated and Real Data
An autonomous robot should be able to evaluate the affordances that are offered by a given situation. Here we address this problem by designing a system that can densely predict affordances given only a single 2D RGB image. This is achieved with a convolutional neural network (ResNet), which we combine with refinement modules recently proposed for addressing semantic image segmentation. We define a novel cost function, which is able to handle (potentially multiple) affordances of objects and their parts in a pixel-wise manner even in the case of incomplete data. We perform qualitative as well as quantitative evaluations with simulated and real data assessing 15 different affordances. In general, we find that affordances, which are well-enough represented in the training data, are correctly recognized with a substantial fraction of correctly assigned pixels. Furthermore, we show that our model outperforms several baselines. Hence, this method can give clear action guidelines for a robot.
Code (0)
등록된 구현이 없습니다.
Tasks
Image SegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows
Understanding how humans interact with the surrounding environment, and specifically reasoning about object interactions and affordances, is a critical challenge in computer vision, robotics, and AI. Current approaches o…
Point CloudsMulti-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordances
Learning object affordances is an effective tool in the field of robot learning. While the data-driven models investigate affordances of single or paired objects, there is a gap in the exploration of affordances of compo…
ObjectEgocentric affordance detection with the one-shot geometry-driven Interaction Tensor
In this abstract we describe recent [4,7] and latest work on the determination of affordances in visually perceived 3D scenes. Our method builds on the hypothesis that geometry on its own provides enough information to e…
Affordance DetectionDetecting Affordances by Visuomotor Simulation
The term "affordance" denotes the behavioral meaning of objects. We propose a cognitive architecture for the detection of affordances in the visual modality. This model is based on the internal simulation of movement seq…
Affordance DetectionLearning Human Activities and Object Affordances from RGB-D Videos
Understanding human activities and object affordances are two very important skills, especially for personal robots which operate in human environments. In this work, we consider the problem of extracting a descriptive l…
DescriptiveObjectSkeleton Based Action Recognition