GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear
Multimodal large language models (LMMs) excel in world knowledge and problem-solving abilities. Through the use of a world-facing camera and contextual AI, emerging smart accessories aim to provide a seamless interface between humans and LMMs. Yet, these wearable computing systems lack an understanding of the user's attention. We introduce GazeGPT as a new user interaction paradigm for contextual AI. GazeGPT uses eye tracking to help the LMM understand which object in the world-facing camera view a user is paying attention to. Using extensive user evaluations, we show that this gaze-contingent mechanism is a faster and more accurate pointing mechanism than alternatives; that it augments human capabilities by significantly improving their accuracy in a dog-breed classification task; and that it is consistently ranked as more natural than head- or body-driven selection mechanisms for contextual AI. Moreover, we prototype a variety of application scenarios that suggest GazeGPT could be of significant value to users as part of future AI-driven personal assistants.
Code (0)
등록된 구현이 없습니다.
Tasks
World KnowledgeSimilar Papers 제목 키워드 기반
Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses
Human eye gaze plays a significant role in many virtual and augmented reality (VR/AR) applications, such as gaze-contingent rendering, gaze-based interaction, or eye-based activity recognition. However, prior works on ga…
Activity RecognitionGaze PredictionHuman-Object Interaction DetectionImage based Eye Gaze Tracking and its Applications
Eye movements play a vital role in perceiving the world. Eye gaze can give a direct indication of the users point of attention, which can be useful in improving human-computer interaction. Gaze estimation in a non-intrus…
Activity RecognitionGaze EstimationHoloFace: Augmenting Human-to-Human Interactions on HoloLens
We present HoloFace, an open-source framework for face alignment, head pose estimation and facial attribute retrieval for Microsoft HoloLens. HoloFace implements two state-of-the-art face alignment methods which can be u…
AttributeEmotion RecognitionFace AlignmentFace Model+3Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding
Human gaze data offer cognitive information that reflects natural language comprehension. Indeed, augmenting language models with human scanpaths has proven beneficial for a range of NLP tasks, including language underst…
Language ModelingLanguage ModellingNatural Language UnderstandingEfficiently Guiding Imitation Learning Agents with Human Gaze
Human gaze is known to be an intention-revealing signal in human demonstrations of tasks. In this work, we use gaze cues from human demonstrators to enhance the performance of agents trained via three popular imitation l…
Atari GamesImitation LearningReinforcement Learning