Mirror-Aware Neural Humans
Human motion capture either requires multi-camera systems or is unreliable when using single-view input due to depth ambiguities. Meanwhile, mirrors are readily available in urban environments and form an affordable alternative by recording two views with only a single camera. However, the mirror setting poses the additional challenge of handling occlusions of real and mirror image. Going beyond existing mirror approaches for 3D human pose estimation, we utilize mirrors for learning a complete body model, including shape and dense appearance. Our main contributions are extending articulated neural radiance fields to include a notion of a mirror, making it sample-efficient over potential occlusion regions. Together, our contributions realize a consumer-level 3D motion capture system that starts from off-the-shelf 2D poses by automatically calibrating the camera, estimating mirror orientation, and subsequently lifting 2D keypoint detections to 3D skeleton pose that is used to condition the mirror-aware NeRF. We empirically demonstrate the benefit of learning a body model and accounting for occlusion in challenging mirror scenes.
Code (1)
Tasks
3D Human Pose EstimationNeRFPose EstimationSimilar Papers 제목 키워드 기반
Let's be friends! A rapport-building 3D embodied conversational agent for the Human Support Robot
Partial subtle mirroring of nonverbal behaviors during conversations (also known as mimicking or parallel empathy), is essential for rapport building, which in turn is essential for optimal human-human communication outc…
Robot self/other distinction: active inference meets neural networks learning in a mirror
Self/other distinction and self-recognition are important skills for interacting with the world, as it allows humans to differentiate own actions from others and be self-aware. However, only a selected group of animals, …
Robot in the mirror: toward an embodied computational model of mirror self-recognition
Self-recognition or self-awareness is a capacity attributed typically only to humans and few other species. The definitions of these concepts vary and little is known about the mechanisms behind them. However, there is a…
Novelty DetectionThe Role of Valence and Meta-awareness in Mirror Self-recognition Using Hierarchical Active Inference
The underlying processes that enable self-perception are crucial for understanding multisensory integration, body perception and action, and the development of the self. Previous computational models have overlooked an e…
MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communication
Communication is a hallmark of intelligence. In this work, we present MIRROR, an approach to (i) quickly learn human models from human demonstrations, and (ii) use the models for subsequent communication planning in assi…
Imitation Learning