CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
This paper focuses on the challenge of answering questions in scenarios that are composed of rich and complex dynamic audio-visual components. Although existing Multimodal Large Language Models (MLLMs) can respond to audio-visual content, these responses are sometimes ambiguous and fail to describe specific audio-visual events. To overcome this limitation, we introduce the CAT, which enhances MLLM in three ways: 1) besides straightforwardly bridging audio and video, we design a clue aggregator that aggregates question-related clues in dynamic audio-visual scenarios to enrich the detailed knowledge required for large language models. 2) CAT is trained on a mixed multimodal dataset, allowing direct application in audio-visual scenarios. Notably, we collect an audio-visual joint instruction dataset named AVinstruct, to further enhance the capacity of CAT to model cross-semantic correlations. 3) we propose AI-assisted ambiguity-aware direct preference optimization, a strategy specialized in retraining the model to favor the non-ambiguity response and improve the ability to localize specific audio-visual objects. Extensive experimental results demonstrate that CAT outperforms existing methods on multimodal tasks, especially in Audio-Visual Question Answering (AVQA) tasks. The codes and the collected instructions are released at https://github.com/rikeilong/Bay-CAT.
Code (1)
Tasks
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Language ModelingLanguage ModellingLarge Language ModelMultimodal Large Language ModelQuestion AnsweringVideo-based Generative Performance BenchmarkingVisual Question AnsweringZero-Shot Video Question AnswerSimilar Papers 제목 키워드 기반
LOVA3: Learning to Visual Question Answering, Asking and Assessment
Question answering, asking, and assessment are three innate human traits crucial for understanding the world and acquiring knowledge. By enhancing these capabilities, humans can more effectively utilize data, leading to …
Question AnsweringVisual Question AnsweringLLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning
Multimodal large language models (LLMs) have achieved notable success across various domains, while research in the medical field has largely focused on unimodal images. Meanwhile, current general-domain multimodal model…
Answer GenerationQuestion-Answer-GenerationQuestion AnsweringVideo Question AnsweringTowards Answering Health-related Questions from Medical Videos: Datasets and Approaches
The increase in the availability of online videos has transformed the way we access information and knowledge. A growing number of individuals now prefer instructional videos as they offer a series of step-by-step proced…
Visual Question Decomposition on Multimodal Large Language Models
Question decomposition has emerged as an effective strategy for prompting Large Language Models (LLMs) to answer complex questions. However, while existing methods primarily focus on unimodal language models, the questio…
Visual Question Answering (VQA)Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
Recent advancements in Chain-of-Thought (CoT) and related rationale-based works have significantly improved the performance of Large Language Models (LLMs) in complex reasoning tasks. With the evolution of Multimodal Lar…
Multimodal ReasoningVisual Question AnsweringVisual Reasoning