HoloFace: Augmenting Human-to-Human Interactions on HoloLens
We present HoloFace, an open-source framework for face alignment, head pose estimation and facial attribute retrieval for Microsoft HoloLens. HoloFace implements two state-of-the-art face alignment methods which can be used interchangeably: one running locally and one running on a remote backend. Head pose estimation is accomplished by fitting a deformable 3D model to the landmarks localized using face alignment. The head pose provides both the rotation of the head and a position in the world space. The parameters of the fitted 3D face model provide estimates of facial attributes such as mouth opening or smile. Together the above information can be used to augment the faces of people seen by the HoloLens user, and thus their interaction. Potential usage scenarios include facial recognition, emotion recognition, eye gaze tracking and many others. We demonstrate the capabilities of our framework by augmenting the faces of people seen through the HoloLens with various objects and animations.
Code (1)
Tasks
AttributeEmotion RecognitionFace AlignmentFace ModelHead Pose EstimationPose EstimationRetrievalSimilar Papers 제목 키워드 기반
HOLa: HoloLens Object Labeling
In the context of medical Augmented Reality (AR) applications, object tracking is a key challenge and requires a significant amount of annotation masks. As segmentation foundation models like the Segment Anything Model (…
ObjectObject TrackingUnityZero Shot SegmentationSemantic Interaction in Augmented Reality Environments for Microsoft HoloLens
Augmented Reality is a promising technique for human-machine interaction. Especially in robotics, which always considers systems in their environment, it is highly beneficial to display visualizations and receive user in…
2D Semantic SegmentationObjectSemantic SegmentationEgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices
Understanding social interactions from egocentric views is crucial for many applications, ranging from assistive robotics to AR/VR. Key to reasoning about interactions is to understand the body pose and motion of the int…
Motion EstimationLearning to Detect Human-Object Interactions
We study the problem of detecting human-object interactions (HOI) in static images, defined as predicting a human and an object bounding box with an interaction class label that connects them. HOI detection is a fundamen…
General ClassificationHuman-Object Interaction DetectionObjectAugmenting Human Cognition through Everyday AR
As spatial computing and multimodal LLMs mature, AR is tending to become an intuitive "thinking tool," embedding semantic and context-aware intelligence directly into everyday environments. This paper explores how always…