Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB
Despite the recent advances in computer vision research, estimating the 3D human pose from single RGB images remains a challenging task, as multiple 3D poses can correspond to the same 2D projection on the image. In this context, depth data could help to disambiguate the 2D information by providing additional constraints about the distance between objects in the scene and the camera. Unfortunately, the acquisition of accurate depth data is limited to indoor spaces and usually is tied to specific depth technologies and devices, thus limiting generalization capabilities. In this paper, we propose a method able to leverage the benefits of depth information without compromising its broader applicability and adaptability in a predominantly RGB-camera-centric landscape. Our approach consists of a heatmap-based 3D pose estimator that, leveraging the paradigm of Privileged Information, is able to hallucinate depth information from the RGB frames given at inference time. More precisely, depth information is used exclusively during training by enforcing our RGB-based hallucination network to learn similar features to a backbone pre-trained only on depth data. This approach proves to be effective even when dealing with limited and small datasets. Experimental results reveal that the paradigm of Privileged Information significantly enhances the model's performance, enabling efficient extraction of depth information by using only RGB images.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Human Pose EstimationHallucinationPose EstimationSimilar Papers 제목 키워드 기반
RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images
This paper proposes a method for hand pose estimation from RGB images that uses both external large-scale depth image datasets and paired depth and RGB images as privileged information at training time. We show that prov…
3D Hand Pose EstimationHand Pose EstimationPose EstimationMasked by Consensus: Disentangling Privileged Knowledge in LLM Correctness
Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers. We investigate whether large language models possess similar privileged knowledge about answer …
Hard Pixel Mining for Depth Privileged Semantic Segmentation
Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to hel…
Depth EstimationDepth PredictionSegmentationSemantic SegmentationSoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience
Parkour poses a significant challenge for legged robots, requiring navigation through complex environments with agility and precision based on limited sensory inputs. In this work, we introduce a novel method for trainin…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Vision-Based Dribbling for Humanoid Soccer via Privileged Representation Learning
Recent advances in humanoid robotics have highlighted the importance of deployable loco-manipulation skills. Dribbling a soccer ball while evading active opponents requires simultaneous balance, precise ball control, and…
Representation LearningReinforcement Learning