Training-Free Action Recognition and Goal Inference with Dynamic Frame Selection
We introduce VidTFS, a Training-free, open-vocabulary video goal and action inference framework that combines the frozen vision foundational model (VFM) and large language model (LLM) with a novel dynamic Frame Selection module. Our experiments demonstrate that the proposed frame selection module improves the performance of the framework significantly. We validate the performance of the proposed VidTFS on four widely used video datasets, including CrossTask, COIN, UCF101, and ActivityNet, covering goal inference and action recognition tasks under open-vocabulary settings without requiring any training or fine-tuning. The results show that VidTFS outperforms pretrained and instruction-tuned multimodal language models that directly stack LLM and VFM for downstream video inference tasks. Our VidTFS with its adaptability shows the future potential for generalizing to new training-free video inference tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionLanguage ModelingLanguage ModellingLarge Language ModelNavigateSimilar Papers 제목 키워드 기반
Human Goal Recognition as Bayesian Inference: Investigating the Impact of Actions, Timing, and Goal Solvability
Goal recognition is a fundamental cognitive process that enables individuals to infer intentions based on available cues. Current goal recognition algorithms often take only observed actions as input, but here we use a B…
Bayesian InferenceSokobanGoal recognition via model-based and model-free techniques
Goal recognition aims at predicting human intentions from a trace of observations. This ability allows people or organizations to anticipate future actions and intervene in a positive (collaborative) or negative (adversa…
modelGoal Recognition as Reinforcement Learning
Most approaches for goal recognition rely on specifications of the possible dynamics of the actor in the environment when pursuing a goal. These specifications suffer from two key issues. First, encoding these dynamics r…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Backprop-Free Reinforcement Learning with Active Neural Generative Coding
In humans, perceptual awareness facilitates the fast recognition and extraction of information from sensory input. This awareness largely depends on how the human agent interacts with the environment. In this work, we pr…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)DataWords: Getting Contrarian with Text, Structured Data and Explanations
Our goal is to build classification models using a combination of free-text and structured data. To do this, we represent structured data by text sentences, DataWords, so that similar data items are mapped into the same …
Classificationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+3