Self-supervised speech representation learning for keyword-spotting with light-weight transformers
Self-supervised speech representation learning (S3RL) is revolutionizing the way we leverage the ever-growing availability of data. While S3RL related studies typically use large models, we employ light-weight networks to comply with tight memory of compute-constrained devices. We demonstrate the effectiveness of S3RL on a keyword-spotting (KS) problem by using transformers with 330k parameters and propose a mechanism to enhance utterance-wise distinction, which proves crucial for improving performance on classification tasks. On the Google speech commands v2 dataset, the proposed method applied to the Auto-Regressive Predictive Coding S3RL led to a 1.2% accuracy improvement compared to training from scratch. On an in-house KS dataset with four different keywords, it provided 6% to 23.7% relative false accept improvement at fixed false reject rate. We argue this demonstrates the applicability of S3RL approaches to light-weight models for KS and confirms S3RL is a powerful alternative to traditional supervised learning for resource-constrained applications.
Code (0)
등록된 구현이 없습니다.
Tasks
Keyword SpottingRepresentation LearningSpeech Representation LearningSimilar Papers 제목 키워드 기반
On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation
Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we pr…
Keyword SpottingKnowledge DistillationRepresentation LearningSpeech Representation LearningMulti-task Voice Activated Framework using Self-supervised Learning
Self-supervised learning methods such as wav2vec 2.0 have shown promising results in learning speech representations from unlabelled and untranscribed speech data that are useful for speech recognition. Since these repre…
Emotion ClassificationKeyword SpottingMulti-Task LearningSelf-Supervised Learning+3Exploring Representation Learning for Small-Footprint Keyword Spotting
In this paper, we investigate representation learning for low-resource keyword spotting (KWS). The main challenges of KWS are limited labeled data and limited available device resources. To address those challenges, we e…
Contrastive LearningKeyword SpottingRepresentation LearningSmall-Footprint Keyword SpottingAttention-Free Keyword Spotting
Till now, attention-based models have been used with great success in the keyword spotting problem domain. However, in light of recent advances in deep learning, the question arises whether self-attention is truly irrepl…
Keyword SpottingImproving Small Footprint Few-shot Keyword Spotting with Supervision on Auxiliary Data
Few-shot keyword spotting (FS-KWS) models usually require large-scale annotated datasets to generalize to unseen target keywords. However, existing KWS datasets are limited in scale and gathering keyword-like labeled dat…
Keyword SpottingMulti-Task LearningSelf-Supervised Learning