One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification

2023-05-27 · Jungwoo Heo, Chan-yeong Lim, Ju-ho Kim, Hyun-seo Shin, Ha-Jin Yu

The application of speech self-supervised learning (SSL) models has achieved remarkable performance in speaker verification (SV). However, there is a computational cost hurdle in employing them, which makes development and deployment difficult. Several studies have simply compressed SSL models through knowledge distillation (KD) without considering the target task. Consequently, these methods could not extract SV-tailored features. This paper suggests One-Step Knowledge Distillation and Fine-Tuning (OS-KDFT), which incorporates KD and fine-tuning (FT). We optimize a student model for SV during KD training to avert the distillation of inappropriate information for the SV. OS-KDFT could downsize Wav2Vec 2.0 based ECAPA-TDNN size by approximately 76.2%, and reduce the SSL model's inference time by 79% while presenting an EER of 0.98%. The proposed OS-KDFT is validated across VoxCeleb1 and VoxCeleb2 datasets and W2V2 and HuBERT SSL models. Experiments are available on our GitHub.

📄 PDF Abstract BibTeX arXiv:2305.17394

Code (1)

jungwoo4021/os-kdft 공식 구현 pytorch

Tasks

Knowledge Distillation Self-Supervised Learning Speaker Verification

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Context Unaware Knowledge Distillation for Image Retrieval

2022-07-19 · Bytasandram Yaswanth Reddy, Shiv Ram Dubey, Rakesh Kumar Sanodiya, Ravi Ranjan Prasad Karn

Existing data-dependent hashing methods use large backbone networks with millions of parameters and are computationally complex. Existing knowledge distillation methods use logits and other features of the deep (teacher)…

Image Retrieval Knowledge Distillation Quantization Retrieval

Distilling Empathy from Large Language Models

2025-07-10 · Henry J. Xie, Jinghan Zhang, Xinhao Zhang, Kunpeng Liu arxiv

The distillation of knowledge from Large Language Models (LLMs) into Smaller Language Models (SLMs), preserving the capabilities and performance of LLMs while reducing model size, has played a key role in the proliferati…

Memorization Dynamics in Knowledge Distillation for Language Models

2026-01-21 · Jaydeep Borkar, Karan Chadha, Niloofar Mireshghallah, Yuchen Zhang 외 arxiv

Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility while often surpassing standard fine-tu…

Knowledge Distillation

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

2023-05-03 · Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost 외

Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning…

Mentor-KD: Making Small Language Models Better Multi-step Reasoners

2024-10-11 · Hojae Lee, Junho Kim, SangKeun Lee

Large Language Models (LLMs) have displayed remarkable performances across various complex tasks by leveraging Chain-of-Thought (CoT) prompting. Recently, studies have proposed a Knowledge Distillation (KD) approach, rea…