paper-with-me

홈 › Papers

MoPD: Mixture-of-Prompts Distillation for Vision-Language Models

2024-12-26 · Yang Chen, Shuai Fu, Yu Zhang

Soft prompt learning methods are effective for adapting vision-language models (VLMs) to downstream tasks. Nevertheless, empirical evidence reveals a tendency of existing methods that they overfit seen classes and exhibit degraded performance on unseen classes. This limitation is due to the inherent bias in the training data towards the seen classes. To address this issue, we propose a novel soft prompt learning method, named Mixture-of-Prompts Distillation (MoPD), which can effectively transfer useful knowledge from hard prompts manually hand-crafted (a.k.a. teacher prompts) to the learnable soft prompt (a.k.a. student prompt), thereby enhancing the generalization ability of soft prompts on unseen classes. Moreover, the proposed MoPD method utilizes a gating network that learns to select hard prompts used for prompt distillation. Extensive experiments demonstrate that the proposed MoPD method outperforms state-of-the-art baselines especially on on unseen classes.

📄 PDF Abstract BibTeX arXiv:2412.19087

Code (0)

등록된 구현이 없습니다.

Tasks

Prompt Learning

Similar Papers 제목 키워드 기반

MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

2026-09-25 · Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu 외 hf

Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This depe…

D^3-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

2026-08-25 · Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li 외 hf

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix th…

Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation

2026-05-26 · Tianlei Chen, Jiao Ou, Ziyuan Liu, Ruiming Tang 외 arxiv

Domain specialization can improve LLM behavior in vertical domains, but often weakens the general capabilities inherited from the original model. Recent Multi-Teacher On-Policy Distillation (MOPD) pipelines recover model…

PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation

2026-09-28 · Youzhi Liu, Ruobing Zheng, Boyuan Tong, Tianqi Li 외 hf

Multi-teacher on-policy distillation (MOPD) has emerged as a popular post-training paradigm for integrating specialized capabilities in frontier language models. Existing OPD research has primarily focused on optimizing …

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

2026-09-27 · Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang 외 hf

Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation …

Knowledge Distillation