Papez: Resource-Efficient Speech Separation with Auditory Working Memory
Transformer-based models recently reached state-of-the-art single-channel speech separation accuracy; However, their extreme computational load makes it difficult to deploy them in resource-constrained mobile or IoT devices. We thus present Papez, a lightweight and computation-efficient single-channel speech separation model. Papez is based on three key techniques. We first replace the inter-chunk Transformer with small-sized auditory working memory. Second, we adaptively prune the input tokens that do not need further processing. Finally, we reduce the number of parameters through the recurrent transformer. Our extensive evaluation shows that Papez achieves the best resource and accuracy tradeoffs with a large margin. We publicly share our source code at \texttt{https://github.com/snuhcs/Papez}
Code (1)
Tasks
Speech SeparationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits
Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active resear…
Speech SeparationA General Close-loop Predictive Coding Framework for Auditory Working Memory
Auditory working memory is essential for various daily activities, such as language acquisition, conversation. It involves the temporary storage and manipulation of information that is no longer present in the environmen…
Language AcquisitionSemantic SimilaritySemantic Textual SimilarityTDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion
Audio-visual speech separation has gained significant traction in recent years due to its potential applications in various fields such as speech recognition, diarization, scene analysis and assistive technologies. Desig…
speech-recognitionSpeech RecognitionSpeech SeparationPac-HuBERT: Self-Supervised Music Source Separation via Primitive Auditory Clustering and Hidden-Unit BERT
In spite of the progress in music source separation research, the small amount of publicly-available clean source data remains a constant limiting factor for performance. Thus, recent advances in self-supervised learning…
ClusteringDecoderMusic Source SeparationSelf-Supervised LearningJointly Learning Visual and Auditory Speech Representations from Raw Data
We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets…
Audio-Visual Speech RecognitionLipreadingspeech-recognitionSpeech Recognition+1