Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement
Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49\%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationSimilar Papers 제목 키워드 기반
Teachers in concordance for pseudo-labeling of 3D sequential data
Automatic pseudo-labeling is a powerful tool to tap into large amounts of sequential unlabeled data. It is specially appealing in safety-critical applications of autonomous driving, where performance requirements are ext…
3D Object Detection3D Semantic SegmentationAutonomous Drivingobject-detection+3TEACHERS RECORD MANAGEMENT SYSTEM PROJECT REPORT.
The Teachers Record Management System is a comprehensive solution designed to simplify the process of managing teacher records. In today's fast-paced world, it is imperative for schools and individuals to have access to …
ManagementNavigateInconsistency Matters: A Knowledge-guided Dual-inconsistency Network for Multi-modal Rumor Detection
Rumor spreaders are increasingly utilizing multimedia content to attract the attention and trust of news consumers. Though a set of rumor detection models have exploited the multi-modal data, they seldom consider the inc…
Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers
Recent approaches have explored language-guided classifiers capable of classifying examples from novel tasks when provided with task-specific natural language explanations, instructions or prompts (Sanh et al., 2022; R. …
Test-time AdaptationTeacher agency in the age of generative AI: towards a framework of hybrid intelligence for learning design
Generative AI (genAI) is being used in education for different purposes. From the teachers' perspective, genAI can support activities such as learning design. However, there is a need to study the impact of genAI on the …