Egocentric Prediction of Action Target in 3D
We are interested in anticipating as early as possible the target location of a person's object manipulation action in a 3D workspace from egocentric vision. It is important in fields like human-robot collaboration, but has not yet received enough attention from vision and learning communities. To stimulate more research on this challenging egocentric vision task, we propose a large multimodality dataset of more than 1 million frames of RGB-D and IMU streams, and provide evaluation metrics based on our high-quality 2D and 3D labels from semi-automatic annotation. Meanwhile, we design baseline methods using recurrent neural networks and conduct various ablation studies to validate their effectiveness. Our results demonstrate that this new task is worthy of further study by researchers in robotics, vision, and learning communities.
Code (0)
등록된 구현이 없습니다.
Tasks
PredictionSimilar Papers 제목 키워드 기반
Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition
We address a novel cross-domain few-shot learning task (CD-FSL) with multimodal input and unlabeled target data for egocentric action recognition. This paper simultaneously tackles two critical challenges associated with…
Action RecognitionCross-Domain Few-Shotcross-domain few-shot learningFew-Shot LearningRobust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning
As the demand for analyzing egocentric videos grows, egocentric visual attention prediction, anticipating where a camera wearer will attend, has garnered increasing attention. However, it remains challenging due to the i…
EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
Egocentric human data offers scalable supervision for robot manipulation. However, behavior cloning entangles transferable content like objects, scenes, and task semantics, with non-transferable factors like human morpho…
Robot ManipulationLearning to Anticipate Egocentric Actions by Imagination
Anticipating actions before they are executed is crucial for a wide range of practical applications, including autonomous driving and robotics. In this paper, we study the egocentric action anticipation task, which predi…
Action AnticipationAutonomous DrivingContrastive LearningUni-Hand: Universal Hand Motion Forecasting in Egocentric Views
Forecasting how human hands move in egocentric views is critical for applications like augmented reality and human-robot policy transfer. Recently, several hand trajectory prediction (HTP) methods have been developed to …
Trajectory PredictionAction AnticipationMotion Forecasting