H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation
On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories. However, existing OPD methods for multimodal reasoning usually rely on a static teacher routing, assigning each sample to a single teacher based on modality or task type. This ignores that visual grounding and abstract reasoning may dominate different decoding steps, making a single teacher insufficient for the full trajectory. To this end, H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning. By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing with token-level teacher arbitration along the shared student trajectory. H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics, and uses a confidence-aware arbitration mechanism to dynamically combine vision-language teacher and text-only teachers at each token. Extensive evaluations over 11 widely-used reasoning benchmarks showcase the superior performance of our method.
Code (0)
등록된 구현이 없습니다.
Tasks
Multimodal ReasoningVisual GroundingSimilar Papers 제목 키워드 기반
Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student
Multimodal sarcasm detection is challenging, especially in low-resource settings where subtle image-text contradictions are hard to learn due to scarce annotated data, which hinders the model's performance. Parameter-eff…
parameter-efficient fine-tuningSarcasm DetectionCPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification
Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-inten…
Video ClassificationMURAL: Multimodal Uncertainty-aware Recommendation via Adaptive edge Learning
Multimodal Graph Neural Networks have become standard for recommendation by augmenting sparse interaction data with content features. Yet current architectures face two bottlenecks: structural rigidity, from a reliance o…
Multimodal RecommendationA Confidence-Aware Multimodal Fusion Framework for Industrial Human-Robot Collaboration
A confidence-aware multimodal fusion framework (CAMF) is proposed to realize reliable human intention prediction for industrial human-robot collaboration. This framework fuses four heterogeneous modalities including obje…
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Knowledge distillation has emerged as a powerful technique for model compression, enabling the transfer of knowledge from large teacher networks to compact student models. However, traditional knowledge distillation meth…
Knowledge DistillationImage ClassificationModel Compression