Asymmetric metric learning for knowledge transfer
Knowledge transfer from large teacher models to smaller student models has recently been studied for metric learning, focusing on fine-grained classification. In this work, focusing on instance-level image retrieval, we study an asymmetric testing task, where the database is represented by the teacher and queries by the student. Inspired by this task, we introduce asymmetric metric learning, a novel paradigm of using asymmetric representations at training. This acts as a simple combination of knowledge transfer with the original metric learning task. We systematically evaluate different teacher and student models, metric learning and knowledge transfer loss functions on the new asymmetric testing as well as the standard symmetric testing task, where database and queries are represented by the same model. We find that plain regression is surprisingly effective compared to more complex knowledge transfer mechanisms, working best in asymmetric testing. Interestingly, our asymmetric metric learning approach works best in symmetric testing, allowing the student to even outperform the teacher.
Code (2)
Tasks
Image RetrievalMetric LearningRetrievalTransfer LearningSimilar Papers 제목 키워드 기반
Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries
Knowledge transfer in multi-task learning is typically viewed as a dichotomy; positive transfer, which improves the performance of all tasks, or negative transfer, which hinders the performance of all tasks. In this pape…
Multi-Task LearningTransfer LearningDeep Asymmetric Multi-task Feature Learning
We propose Deep Asymmetric Multitask Feature Learning (Deep-AMTFL) which can learn deep representations shared across multiple tasks while effectively preventing negative transfer that may happen in the feature sharing p…
image-classificationImage ClassificationTransfer LearningTemporal Probabilistic Asymmetric Multi-task Learning
When performing multi-task predictions with time-series data, knowledge learned for one task at a specific time step may be useful in learning for another task at a later time step (e.g. prediction of sepsis may be usefu…
Multi-Task LearningPredictionTime SeriesTime Series Analysis+1Cross-Lingual Transfer of Cultural Knowledge: An Asymmetric Phenomenon
Despite substantial research efforts evaluating how well large language models~(LLMs) handle global cultural diversity, the mechanisms behind their cultural knowledge acquisition, particularly in multilingual settings, r…
Cross-Lingual TransferDiversityClinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Learning
Although recent multi-task learning methods have shown to be effective in improving the generalization of deep neural networks, they should be used with caution for safety-critical applications, such as clinical risk pre…
Knowledge GraphsMulti-Task LearningPredictionTime Series+3