Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation
Continual learning focuses on incrementally training a model on a sequence of tasks with the aim of learning new tasks while minimizing performance drop on previous tasks. Existing approaches at the intersection of Continual Learning and Visual Question Answering (VQA) do not study how the multimodal nature of the input affects the learning dynamics of a model. In this paper, we demonstrate that each modality evolves at different rates across a continuum of tasks and that this behavior occurs in established encoder-only models as well as modern recipes for developing Vision & Language (VL) models. Motivated by this observation, we propose a modality-aware feature distillation (MAFED) approach which outperforms existing baselines across models of varying scale in three multimodal continual learning settings. Furthermore, we provide ablations showcasing that modality-aware distillation complements experience replay. Overall, our results emphasize the importance of addressing modality-specific dynamics to prevent forgetting in multimodal continual learning.
Code (1)
Tasks
Continual LearningQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Continual Visual Question Answering (CVQA) based on pre-trained models(PTMs) has achieved promising progress by leveraging prompt tuning to enable continual multi-modal learning. However, most existing methods adopt cros…
Continual LearningQuestion AnsweringVisual Question AnsweringAIM: Asymmetric Information Masking for Visual Question Answering Continual Learning
In continual visual question answering (VQA), existing Continual Learning (CL) methods are mostly built for symmetric, unimodal architectures. However, modern Vision-Language Models (VLMs) violate this assumption, as the…
Visual Question AnsweringContinual LearningTask Formulation Matters When Learning Continuously: A Case Study in Visual Question Answering
Continual learning is a promising alternative to the current pretrain-and-finetune paradigm: It aims to learn a model on a sequence of tasks without forgetting knowledge from preceding tasks. We investigate continual lea…
Continual LearningIncremental LearningQuestion AnsweringSemantic Similarity+3CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
Multimodal large language models (MLLMs) have garnered widespread attention from researchers due to their remarkable understanding and generation capabilities in visual language tasks (e.g., visual question answering). H…
Continual LearningLanguage ModelingLanguage ModellingLarge Language Model+5AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning
In this paper, a novel benchmark for audio-visual question answering continual learning (AVQACL) is introduced, aiming to study fine-grained scene understanding and spatial-temporal reasoning in videos under a contin…
Audio-visual Question AnsweringContinual LearningKnowledge DistillationQuestion Answering+2