WAYLA - Generating Images from Eye Movements
We present a method for reconstructing images viewed by observers based only on their eye movements. By exploring the relationships between gaze patterns and image stimuli, the "What Are You Looking At?" (WAYLA) system learns to synthesize photo-realistic images that are similar to the original pictures being viewed. The WAYLA approach is based on the Conditional Generative Adversarial Network (Conditional GAN) image-to-image translation technique of Isola et al. We consider two specific applications - the first, of reconstructing newspaper images from gaze heat maps, and the second, of detailed reconstruction of images containing only text. The newspaper image reconstruction process is divided into two image-to-image translation operations, the first mapping gaze heat maps into image segmentations, and the second mapping the generated segmentation into a newspaper image. We validate the performance of our approach using various evaluation metrics, along with human visual inspection. All results confirm the ability of our network to perform image generation tasks using eye tracking data.
Code (0)
등록된 구현이 없습니다.
Tasks
Generative Adversarial NetworkImage GenerationImage ReconstructionImage-to-Image TranslationTranslationSimilar Papers 제목 키워드 기반
Motion Transfer-Enhanced StyleGAN for Generating Diverse Macaque Facial Expressions
Generating animal faces using generative AI techniques is challenging because the available training images are limited both in quantity and variation, particularly for facial expressions across individuals. In this stud…
Data AugmentationImage EditingLip Movements Generation at a Glance
Cross-modality generation is an emerging topic that aims to synthesize data in one modality based on information in a different modality. In this paper, we consider a task of such: given an arbitrary audio speech and one…
DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering
While recent advances in deep neural networks have made it possible to render high-quality images, generating photo-realistic and personalized talking head remains challenging. With given audio, the key to tackling this …
NeRFNeural RenderingTalking Head GenerationVersatile Multimodal Controls for Expressive Talking Human Animation
In filmmaking, directors typically allow actors to perform freely based on the script before providing specific guidance on how to present key actions. AI-generated content faces similar requirements, where users not onl…
Human AnimationStyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3
Realistic generative face video synthesis has long been a pursuit in both computer vision and graphics community. However, existing face video generation methods tend to produce low-quality frames with drifted facial ide…
Image GenerationVideo Generation