Triplet Knowledge Distillation
In Knowledge Distillation, the teacher is generally much larger than the student, making the solution of the teacher likely to be difficult for the student to learn. To ease the mimicking difficulty, we introduce a triplet knowledge distillation mechanism named TriKD. Besides teacher and student, TriKD employs a third role called anchor model. Before distillation begins, the pre-trained anchor model delimits a subspace within the full solution space of the target problem. Solutions within the subspace are expected to be easy targets that the student could mimic well. Distillation then begins in an online manner, and the teacher is only allowed to express solutions within the aforementioned subspace. Surprisingly, benefiting from accurate but easy-to-mimic hints, the student can finally perform well. After the student is well trained, it can be used as the new anchor for new students, forming a curriculum learning strategy. Our experiments on image classification and face recognition with various models clearly demonstrate the effectiveness of our method. Furthermore, the proposed TriKD is also effective in dealing with the overfitting issue. Moreover, our theoretical analysis supports the rationality of our triplet distillation.
Code (0)
등록된 구현이 없습니다.
Tasks
Face Recognitionimage-classificationImage ClassificationKnowledge DistillationTripletMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Teacher-Student Training and Triplet Loss to Reduce the Effect of Drastic Face Occlusion
We study a series of recognition tasks in two realistic scenarios requiring the analysis of faces under strong occlusion. On the one hand, we aim to recognize facial expressions of people wearing Virtual Reality (VR) hea…
Age EstimationFacial Expression RecognitionFacial Expression Recognition (FER)Knowledge Distillation+1Triplet Loss for Knowledge Distillation
In recent years, deep learning has spread rapidly, and deeper, larger models have been proposed. However, the calculation cost becomes enormous as the size of the models becomes larger. Various techniques for compressing…
Knowledge DistillationMetric LearningTripletTeacher-Student Training and Triplet Loss for Facial Expression Recognition under Occlusion
In this paper, we study the task of facial expression recognition under strong occlusion. We are particularly interested in cases where 50% of the face is occluded, e.g. when the subject wears a Virtual Reality (VR) head…
Facial Expression RecognitionFacial Expression Recognition (FER)Knowledge DistillationTripletDistilling GANs with Style-Mixed Triplets for X2I Translation with Limited Data
Conditional image synthesis is an integral part of many X2I translation systems, including image-to-image, text-to-image and audio-to-image translation systems. Training these large systems generally requires huge amount…
Image GenerationKnowledge DistillationSemantic SegmentationTranslationA Recipe for Efficient SBIR Models: Combining Relative Triplet Loss with Batch Normalization and Knowledge Distillation
Sketch-Based Image Retrieval (SBIR) is a crucial task in multimedia retrieval, where the goal is to retrieve a set of images that match a given sketch query. Researchers have already proposed several well-performing solu…
Data AugmentationImage RetrievalKnowledge DistillationRetrieval+2