Grounded Human-Object Interaction Hotspots from Video
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object interaction "hotspots" directly from video. Rather than treat affordances as a manually supervised semantic segmentation task, our approach learns about interactions by watching videos of real human behavior and anticipating afforded actions. Given a novel image or video, our model infers a spatial hotspot map indicating how an object would be manipulated in a potential interaction-- even if the object is currently at rest. Through results with both first and third person video, we show the value of grounding affordances in real human-object interactions. Not only are our weakly supervised hotspots competitive with strongly supervised affordance methods, but they can also anticipate object interaction for novel object categories.
Code (1)
Tasks
Human-Object Interaction DetectionObjectObject RecognitionSemantic SegmentationVideo-to-image Affordance GroundingSimilar Papers 제목 키워드 기반
Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object…
Human-Object Interaction DetectionObjectSemantic SegmentationJoint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos
We propose to forecast future hand-object interactions given an egocentric video. Instead of predicting action labels or pixels, we directly predict the hand motion trajectory and the future contact points on the next ac…
ObjectForecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video
We address the challenging task of anticipating human-object interaction in first person videos. Most existing methods ignore how the camera wearer interacts with the objects, or simply consider body motion as a separate…
Action AnticipationHuman-Object Interaction DetectionBidirectional Progressive Transformer for Interaction Intention Anticipation
Interaction intention anticipation aims to jointly predict future hand trajectories and interaction hotspots. Existing research often treated trajectory forecasting and interaction hotspots prediction as separate tasks o…
PredictionTrajectory ForecastingMaking Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars
Generating talking avatars is a fundamental task in video generation. Although existing methods can generate full-body talking avatars with simple human motion, extending this task to grounded human-object interaction (G…
Video Generation