SpecTracle: Wearable Facial Motion Tracking from Unobtrusive Peripheral Cameras
Facial motion tracking in head-mounted displays (HMD) has the potential to enable immersive "face-to-face" interaction in a virtual environment. However, current works on facial tracking are not suitable for unobtrusive augmented reality (AR) glasses or do not have the ability to track arbitrary facial movements. In this work, we demonstrate a novel system called SpecTracle that tracks a user's facial motions using two wide-angle cameras mounted right next to the visor of a Hololens. Avoiding the usage of cameras extended in front of the face, our system greatly improves the feasibility to integrate full-face tracking into a low-profile form factor. We also demonstrate that a neural network-based model processing the wide-angle cameras can run in real-time at 24 frames per second (fps) on a mobile GPU and track independent facial movement for different parts of the face with a user-independent model. Using a short personalized calibration, the system improves its tracking performance by 42.3% compared to the user-independent model.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUSimilar Papers 제목 키워드 기반
Unconstrained Realtime Facial Performance Capture
We introduce a realtime facial tracking system specifically designed for performance capture in unconstrained settings using a consumer-level RGB-D sensor. Our framework provides uninterrupted 3D facial tracking, even in…
Emotion Recognition in Older Adults with Quantum Machine Learning and Wearable Sensors
We investigate the feasibility of inferring emotional states exclusively from physiological signals, thereby presenting a privacy-preserving alternative to conventional facial recognition techniques. We conduct a perform…
Quantum Machine LearningEmotion RecognitionInMyFace: Inertial and Mechanomyography-Based Sensor Fusion for Wearable Facial Activity Recognition
Recognizing facial activity is a well-understood (but non-trivial) computer vision problem. However, reliable solutions require a camera with a good view of the face, which is often unavailable in wearable settings. Furt…
Activity RecognitionSensor FusionContactless pulse rate assessment: Results and insights for application in driving simulator
Camera-based monitoring of Pulse Rate (PR) enables continuous and unobtrusive assessment of driver's state, allowing estimation of fatigue or stress that could impact traffic safety. Commonly used wearable Photoplethysmo…
Photoplethysmography (PPG)What's on Your Plate? Inferring Chinese Cuisine Intake from Wearable IMUs
Accurate food intake detection is vital for dietary monitoring and chronic disease prevention. Traditional self-report methods are prone to recall bias, while camera-based approaches raise concerns about privacy. Further…