MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer

2024-10-14 · Minghao Zhu, Zhengpu Wang, Mengxian Hu, Ronghao Dang, Xiao Lin, Xun Zhou, Chengju Liu, Qijun Chen

Transferring visual-language knowledge from large-scale foundation models for video recognition has proved to be effective. To bridge the domain gap, additional parametric modules are added to capture the temporal information. However, zero-shot generalization diminishes with the increase in the number of specialized parameters, making existing works a trade-off between zero-shot and close-set performance. In this paper, we present MoTE, a novel framework that enables generalization and specialization to be balanced in one unified model. Our approach tunes a mixture of temporal experts to learn multiple task views with various degrees of data fitting. To maximally preserve the knowledge of each expert, we propose \emph{Weight Merging Regularization}, which regularizes the merging process of experts in weight space. Additionally with temporal feature modulation to regularize the contribution of temporal feature during test. We achieve a sound balance between zero-shot and close-set video recognition tasks and obtain state-of-the-art or competitive results on various datasets, including Kinetics-400 \& 600, UCF, and HMDB. Code is available at \url{https://github.com/ZMHH-H/MoTE}.

📄 PDF Abstract BibTeX arXiv:2410.10589

Code (1)

zmhh-h/mote 공식 구현 pytorch

Tasks

Transfer LearningVideo RecognitionZero-shot Generalization

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

2026-05-27 · Yuting Ma, Lechao Cheng, Xiaohua Xu arxiv

Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation un…

Reinforcement Learning Federated Learning

Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning

2024-11-17 · Wenke Huang, Jian Liang, Zekun Shi, Didi Zhu 외

Multimodal Large Language Model (MLLM) have demonstrated strong generalization capabilities across diverse distributions and tasks, largely due to extensive pre-training datasets. Fine-tuning MLLM has become a common pra…

Image Captioning Language Modeling Language Modelling Large Language Model+3

Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs

2026-07-24 · Yuheng Zong, Minghua Wang, Xin Zhao, Zhi-Hui Zhan 외 arxiv

Remote sensing multimodal large language models (RS-MLLMs) have improved general aerial-image understanding. However, Earth observation applications require fine-grained scenario specialization, constrained by scarce hig…

Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization

2022-02-02 · Dianbo Liu, Alex Lamb, Xu Ji, Pascal Notsawo 외

Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit. It has been theoretically and empirically shown that discretization of representation…

Quantization reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Massively Multilingual Lexical Specialization of Multilingual Transformers

2022-08-01 · Tommaso Green, Simone Paolo Ponzetto, Goran Glavaš

While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high-…

Bilingual Lexicon Induction Retrieval Sentence Sentence Retrieval+3