Recycle Your Wav2Vec2 Codebook: A Speech Perceiver for Keyword Spotting
Speech information in a pretrained wav2vec2.0 model is usually leveraged through its encoder, which has at least 95M parameters, being not so suitable for small footprint Keyword Spotting. In this work, we show an efficient way of profiting from wav2vec2.0’s linguistic knowledge, by recycling the phonetic information encoded in its latent codebook, which has been typically thrown away after pretraining. We do so by transferring the codebook as weights for the latent bottleneck of a Keyword Spotting Perceiver, thus initializing such model with phonetic embeddings already. The Perceiver design relies on cross-attention between these embeddings and input data to generate better representations. Our method delivers accuracy gains compared to random initialization, at no latency costs. Plus, we show that the phonetic embeddings can easily be downsampled with k-means clustering, speeding up inference in 3.5 times at only slight accuracy penalties.
Code (0)
등록된 구현이 없습니다.
Tasks
Keyword SpottingSmall-Footprint Keyword SpottingSimilar Papers 제목 키워드 기반
Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices
Open-vocabulary keyword spotting (KWS) refers to the task of detecting words or terms within speech recordings, regardless of whether they were included in the training data. This paper introduces an open-vocabulary keyw…
Keyword SpottingProposal-based Few-shot Sound Event Detection for Speech and Environmental Sounds with Perceivers
Many applications involve detecting and localizing specific sound events within long, untrimmed documents, including keyword spotting, medical observation, and bioacoustic monitoring for conservation. Deep learning techn…
Event DetectionKeyword SpottingRegion ProposalSound Event DetectionOn-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation
Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we pr…
Keyword SpottingKnowledge DistillationRepresentation LearningSpeech Representation LearningMedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capa…
Medical Image SegmentationEfficient Speech Translation with Dynamic Latent Perceivers
Transformers have been the dominant architecture for Speech Translation in recent years, achieving significant improvements in translation quality. Since speech signals are longer than their textual counterparts, and due…
Speech-to-TextSpeech-to-Text TranslationTranslation