Towards Compatible Fine-tuning for Vision-Language Model Updates

2024-12-30 · Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, Tieniu Tan

So far, efficient fine-tuning has become a popular strategy for enhancing the capabilities of foundation models on downstream tasks by learning plug-and-play modules. However, existing methods overlook a crucial issue: if the underlying foundation model is updated, are these plug-and-play modules still effective? In this paper, we first conduct a detailed analysis of various fine-tuning methods on the CLIP in terms of their compatibility with model updates. The study reveals that many high-performing fine-tuning methods fail to be compatible with the upgraded models. To address this, we propose a novel approach, Class-conditioned Context Optimization (ContCoOp), which integrates learnable prompts with class embeddings using an attention layer before inputting them into the text encoder. Consequently, the prompts can dynamically adapt to the changes in embedding space (due to model updates), ensuring continued effectiveness. Extensive experiments over 15 datasets show that our ContCoOp achieves the highest compatibility over the baseline methods, and exhibits robust out-of-distribution generalization.

📄 PDF Abstract BibTeX arXiv:2412.20895

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingOut-of-Distribution Generalization

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models

2026-04-22 · Yucheng Zhou, Junwei Sheng, Qianning Wang, Jianbing Shen arxiv

Supervised Fine-Tuning (SFT) is the predominant paradigm for aligning large language models (LLMs), yet it suffers from optimization instability and limited generalization. Recent work attributes this issue to pathologic…

Reinforcement Learning

Two to Tango: Coupled Task-Reference Selection for Safe LLM Fine-tuning

2026-06-01 · Xinrui Chen, Jianhao Zhang, Ou Wu, Di Gao arxiv

Fine-tuning safety aligned large language models (LLMs) on downstream data improves adaptation but may erode learned safety behavior. Existing methods use fixed safety examples, global constraints, or one-sided task filt…

Continual Learning

Localized Latent Updates for Fine-Tuning Vision-Language Models

2022-12-13 · Moritz Ibing, Isaak Lim, Leif Kobbelt

Although massive pre-trained vision-language models like CLIP show impressive generalization capabilities for many tasks, still it often remains necessary to fine-tune them for improved performance on specific datasets. …

Few-Shot Learning

RandLoRA: Full-rank parameter-efficient fine-tuning of large models

2025-02-03 · Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo 외

Low-Rank Adaptation (LoRA) and its variants have shown impressive results in reducing the number of trainable parameters and memory requirements of large transformer networks while maintaining fine-tuning performance. Th…

parameter-efficient fine-tuning

LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models

2025-08-17 · Krishna Teja Chitty-Venkata, Murali Emani, Venkatram Vishwanath arxiv

Vision Language Models (VLMs) integrate visual and text modalities to enable multimodal understanding and generation. These models typically combine a Vision Transformer (ViT) as an image encoder and a Large Language Mod…

Neural Architecture Search Computational Efficiency Text Generation