Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention
Humans utilize their gaze to concentrate on essential information while perceiving and interpreting intentions in videos. Incorporating human gaze into computational algorithms can significantly enhance model performance in video understanding tasks. In this work, we address a challenging and innovative task in video understanding: predicting the actions of an agent in a video based on a partial video. We introduce the Gaze-guided Action Anticipation algorithm, which establishes a visual-semantic graph from the video input. Our method utilizes a Graph Neural Network to recognize the agent's intention and predict the action sequence to fulfill this intention. To assess the efficiency of our approach, we collect a dataset containing household activities generated in the VirtualHome environment, accompanied by human gaze data of viewing videos. Our method outperforms state-of-the-art techniques, achieving a 7\% improvement in accuracy for 18-class intention recognition. This highlights the efficiency of our method in learning important features from human gaze data.
Code (0)
등록된 구현이 없습니다.
Tasks
Action AnticipationGraph Neural NetworkIntent DetectionVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding
Egocentric action understanding is often addressed using large video models pretrained on extensive exocentric datasets. However, many first-person actions depend on a small number of hand-object interactions involving o…
Action UnderstandingAction RecognitionPredicting the Next Action by Modeling the Abstract Goal
The problem of anticipating human actions is an inherently uncertain one. However, we can reduce this uncertainty if we have a sense of the goal that the actor is trying to achieve. Here, we present an action anticipatio…
Action AnticipationDiffAnt: Diffusion Models for Action Anticipation
Anticipating future actions is inherently uncertain. Given an observed video segment containing ongoing actions, multiple subsequent actions can plausibly follow. This uncertainty becomes even larger when predicting far …
Action AnticipationDeep Future Gaze: Gaze Anticipation on Egocentric Videos Using Adversarial Networks
We introduce a new problem of gaze anticipation on egocentric videos. This substantially extends the conventional gaze prediction problem to future frames by no longer confining it on the current frame. To solve this pro…
Gaze PredictionSemantically Guided Representation Learning For Action Anticipation
Action anticipation is the task of forecasting future activity from a partially observed sequence of events. However, this task is exposed to intrinsic future uncertainty and the difficulty of reasoning upon interconnect…
Action AnticipationRepresentation Learning