HCQA @ Ego4D EgoSchema Challenge 2024
In this report, we present our champion solution for Ego4D EgoSchema Challenge in CVPR 2024. To deeply integrate the powerful egocentric captioning model and question reasoning model, we propose a novel Hierarchical Comprehension scheme for egocentric video Question Answering, named HCQA. It consists of three stages: Fine-grained Caption Generation, Context-driven Summarization, and Inference-guided Answering. Given a long-form video, HCQA captures local detailed visual information and global summarised visual information via Fine-grained Caption Generation and Context-driven Summarization, respectively. Then in Inference-guided Answering, HCQA utilizes this hierarchical information to reason and answer given question. On the EgoSchema blind test set, HCQA achieves 75% accuracy in answering over 5,000 human curated multiple-choice questions. Our code will be released at https://github.com/Hyu-Zhang/HCQA.
Code (1)
Tasks
Caption GenerationEgoSchemaMultiple-choiceQuestion AnsweringVideo Question AnsweringSimilar Papers 제목 키워드 기반
HCQA: Hybrid Classical-Quantum Agent for Generating Optimal Quantum Sensor Circuits
This study proposes an HCQA for designing optimal Quantum Sensor Circuits (QSCs) to address complex quantum physics problems. The HCQA integrates computational intelligence techniques by leveraging a Deep Q-Network (DQN)…
HCqa: Hybrid and Complex Question Answering on Textual Corpus and Knowledge Graph
Question Answering (QA) systems provide easy access to the vast amount of knowledge without having to know the underlying complex structure of the knowledge. The research community has provided ad hoc solutions to the ke…
Knowledge Graphsnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+3VDMA: Video Question Answering with Dynamically Generated Multi-Agents
This technical report provides a detailed description of our approach to the EgoSchema Challenge 2024. The EgoSchema Challenge aims to identify the most appropriate responses to questions regarding a given video clip. In…
EgoSchemaQuestion AnsweringResponse GenerationVideo Question AnsweringEgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of …
DiagnosticEgoSchemaFormMultiple-choice+4Transformers for Complex Query Answering over Knowledge Hypergraphs
Complex Query Answering (CQA) has been extensively studied in recent years. In order to model data that is closer to real-world distribution, knowledge graphs with different modalities have been introduced. Triple KGs, a…
Complex Query AnsweringKnowledge GraphsNegation