On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation

2023-07-06 · Gene-Ping Yang, Yue Gu, Qingming Tang, Dongsu Du, Yuzong Liu

Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we proposed a knowledge distillation-based self-supervised speech representation learning (S3RL) architecture for on-device keyword spotting. Our approach used a teacher-student framework to transfer knowledge from a larger, more complex model to a smaller, light-weight model using dual-view cross-correlation distillation and the teacher's codebook as learning objectives. We evaluated our model's performance on an Alexa keyword spotting detection task using a 16.6k-hour in-house dataset. Our technique showed exceptional performance in normal and noisy conditions, demonstrating the efficacy of knowledge distillation methods in constructing self-supervised models for keyword spotting tasks while working within on-device resource constraints.

📄 PDF Abstract BibTeX arXiv:2307.02720

Code (0)

등록된 구현이 없습니다.

Tasks

Keyword Spotting Knowledge Distillation Representation LearningSpeech Representation Learning

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Self-supervised speech representation learning for keyword-spotting with light-weight transformers

2023-03-07 · Chenyang Gao, Yue Gu, Francesco Caliva, Yuzong Liu

Self-supervised speech representation learning (S3RL) is revolutionizing the way we leverage the ever-growing availability of data. While S3RL related studies typically use large models, we employ light-weight networks t…

Keyword Spotting Representation Learning Speech Representation Learning

SPADE: Self-supervised Pretraining for Acoustic DisEntanglement

2023-02-03 · John Harvill, Jarred Barber, Arun Nair, Ramin Pishehvar

Self-supervised representation learning approaches have grown in popularity due to the ability to train models on large amounts of unlabeled data and have demonstrated success in diverse fields such as natural language p…

Disentanglement Representation Learning Rhythm

On the Impact of Quantization and Pruning of Self-Supervised Speech Models for Downstream Speech Recognition Tasks "In-the-Wild''

2023-09-25 · Arthur Pimentel, Heitor Guimarães, Anderson R. Avila, Mehdi Rezagholizadeh 외

Recent advances with self-supervised learning have allowed speech recognition systems to achieve state-of-the-art (SOTA) word error rates (WER) while requiring only a fraction of the labeled training data needed by its p…

Data Augmentation Model Compression Quantization Self-Supervised Learning+2

On-device Streaming Discrete Speech Units

2025-06-02 · Kwanghee Choi, Masao Someki, Emma Strubell, Shinji Watanabe

Discrete speech units (DSUs) are derived from clustering the features of self-supervised speech models (S3Ms). DSUs offer significant advantages for on-device streaming speech applications due to their rich phonetic info…

RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations

2023-07-03 · Neha Sahipjohn, Neil Shah, Vishal Tambrahalli, Vineet Gandhi

Significant progress has been made in speaker dependent Lip-to-Speech synthesis, which aims to generate speech from silent videos of talking faces. Current state-of-the-art approaches primarily employ non-autoregressive …

Lip to Speech Synthesis Speaker-Specific Lip to Speech Synthesis Speech Synthesis