DeePoint: Visual Pointing Recognition and Direction Estimation
In this paper, we realize automatic visual recognition and direction estimation of pointing. We introduce the first neural pointing understanding method based on two key contributions. The first is the introduction of a first-of-its-kind large-scale dataset for pointing recognition and direction estimation, which we refer to as the DP Dataset. DP Dataset consists of more than 2 million frames of 33 people pointing in various styles annotated for each frame with pointing timings and 3D directions. The second is DeePoint, a novel deep network model for joint recognition and 3D direction estimation of pointing. DeePoint is a Transformer-based network which fully leverages the spatio-temporal coordination of the body parts, not just the hands. Through extensive experiments, we demonstrate the accuracy and efficiency of DeePoint. We believe DP Dataset and DeePoint will serve as a sound foundation for visual human intention understanding.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
We address Embodied Reference Understanding, the task of predicting the object a person in the scene refers to through pointing gesture and language. This requires multimodal reasoning over text, visual pointing cues, an…
Multimodal ReasoningPoint Anywhere: Directed Object Estimation from Omnidirectional Images
One of the intuitive instruction methods in robot navigation is a pointing gesture. In this study, we propose a method using an omnidirectional camera to eliminate the user/object position constraint and the left/right c…
Objectobject-detectionObject DetectionPosition+1Point and Ask: Incorporating Pointing into Visual Question Answering
Visual Question Answering (VQA) has become one of the key benchmarks of visual recognition progress. Multiple VQA extensions have been explored to better simulate real-world settings: different question formulations, cha…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Hand Gesture Recognition Library
In this paper we have presented a hand gesture recognition library. Various functions include detecting cluster count, cluster orientation, finger pointing direction, etc. To use these functions first the input image nee…
Gesture RecognitionHand Gesture RecognitionHand-Gesture RecognitionPointing-Based Object Recognition
This paper presents a comprehensive pipeline for recognizing objects targeted by human pointing gestures using RGB images. As human-robot interaction moves toward more intuitive interfaces, the ability to identify target…
Monocular Depth EstimationObject RecognitionObject DetectionImage Captioning