Characterising Topic Familiarity and Query Specificity Using Eye-Tracking Data
Eye-tracking data has been shown to correlate with a user's knowledge level and query formulation behaviour. While previous work has focused primarily on eye gaze fixations for attention analysis, often requiring additional contextual information, our study investigates the memory-related cognitive dimension by relying solely on pupil dilation and gaze velocity to infer users' topic familiarity and query specificity without needing any contextual information. Using eye-tracking data collected via a lab user study (N=18), we achieved a Macro F1 score of 71.25% for predicting topic familiarity with a Gradient Boosting classifier, and a Macro F1 score of 60.54% with a k-nearest neighbours (KNN) classifier for query specificity. Furthermore, we developed a novel annotation guideline -- specifically tailored for question answering -- to manually classify queries as Specific or Non-specific. This study demonstrates the feasibility of eye-tracking to better understand topic familiarity and query specificity in search.
Code (1)
Tasks
Pupil DilationQuestion AnsweringSpecificityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CIRCLE: Multi-Turn Query Clarifications with Reinforcement Learning
Users often have trouble formulating their information needs into words on the first try when searching online. This can lead to frustration, as they may have to reformulate their queries when retrieved information is no…
Language ModelingLanguage Modellingreinforcement-learningReinforcement Learning+1Behavioral Biometrics for Automatic Detection of User Familiarity in VR
As virtual reality (VR) devices become increasingly integrated into everyday settings, a growing number of users without prior experience will engage with VR systems. Automatically detecting a user's familiarity with VR …
Cognitively Biased Users Interacting with Algorithmically Biased Results in Whole-Session Search on Debated Topics
When interacting with information retrieval (IR) systems, users, affected by confirmation biases, tend to select search results that confirm their existing beliefs on socially significant contentious issues. To understan…
Information RetrievalRetrievalSession SearchToward a Gricean Retreat: Probing LLMs for Knowledge Boundaries and Referent Specificity
When asked about entities outside their knowledge boundary, LLMs routinely fabricate plausible-sounding details rather than backing off to safer, more general claims. We frame this failure through a Gricean lens: a coope…
AwesomeLit: Towards Hypothesis Generation with Agent-Supported Literature Research
There are different goals for literature research, from understanding an unfamiliar topic to generate hypothesis for the next research project. The nature of literature research also varies according to user's familiarit…
Semantic Similarity