Distilling Knowledge From a Deep Pose Regressor Network
This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on "dark knowledge" for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time.
Code (0)
등록된 구현이 없습니다.
Tasks
PredictionregressionTransfer LearningVisual OdometrySimilar Papers 제목 키워드 기반
Distilling Word Embeddings: An Encoding Approach
Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restr…
Word EmbeddingsDistilling HuBERT with LSTMs via Decoupled Knowledge Distillation
Much research effort is being applied to the task of compressing the knowledge of self-supervised models, which are powerful, yet large and memory consuming. In this work, we show that the original method of knowledge di…
Automatic Speech RecognitionKnowledge Distillationspeech-recognitionSpeech RecognitionMasked Autoencoders Enable Efficient Knowledge Distillers
This paper studies the potential of distilling knowledge from pre-trained models, especially Masked Autoencoders. Our approach is simple: in addition to optimizing the pixel reconstruction loss on masked inputs, we minim…
Knowledge DistillationI-DREM MRAC with Time-Varying Adaptation Rate & No A Priori Knowledge of Control Input Matrix Sign to Relax PE Condition
The known dynamic regressor extension and mixing method (DREM) is combined with the proposed filter of a new type, which uses the integration operation with forgetting, and the recursive least-squares method to develop t…
Adversarial Bi-Regressor Network for Domain Adaptive Regression
Domain adaptation (DA) aims to transfer the knowledge of a well-labeled source domain to facilitate unlabeled target learning. When turning to specific tasks such as indoor (Wi-Fi) localization, it is essential to learn …
Domain Adaptationregression