GazBy: Gaze-Based BERT Model to Incorporate Human Attention in Neural Information Retrieval
This paper is interested in investigating whether human gaze signals can be leveraged to improve state-of-the-art search engine performance and how to incorporate this new input signal marked by human attention into existing neural retrieval models. In this paper, we propose GazBy ({\bf Gaz}e-based {\bf B}ert model for document relevanc{\bf y}), a light-weight joint model that integrates human gaze fixation estimation into transformer models to predict document relevance, incorporating more nuanced information about cognitive processing into information retrieval (IR). We evaluate our model on the Text Retrieval Conference (TREC) Deep Learning (DL) 2019 and 2020 Tracks. Our experiments show encouraging results and illustrate the effective and ineffective entry points for using human gaze to help with transformer-based neural retrievers. With the rise of virtual reality (VR) and augmented reality (AR), human gaze data will become more available. We hope this work serves as a first step exploring using gaze signals in modern neural search engines.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalRetrievalText RetrievalSimilar Papers 제목 키워드 기반
Supervising Neural Attention Models for Video Captioning by Human Gaze Data
The attention mechanisms in deep neural networks are inspired by human's attention that sequentially focuses on the most relevant parts of the information over time to generate prediction output. The attention parameters…
DescriptiveGaze PredictionSentenceVideo CaptioningEvaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations
Rationales in the form of manually annotated input spans usually serve as ground truth when evaluating explainability methods in NLP. They are, however, time-consuming and often biased by the annotation process. In this …
validEyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization
Code summarization is the task of generating natural language descriptions of source code, which is critical for software comprehension and maintenance. While large language models (LLMs) have achieved remarkable progres…
Gaze-Regularized VLMs for Ego-Centric Behavior Understanding
Eye gaze, encompassing fixations and saccades, provides critical insights into human intentions and future actions. This study introduces a gaze-regularized framework that enhances Vision Language Models (VLMs) for egoce…
GazeDETR: Gaze Detection using Disentangled Head and Gaze Representations
Gaze communication plays a crucial role in daily social interactions. Quantifying this behavior can help in human-computer interaction and digital phenotyping. While end-to-end models exist for gaze target detection, the…