paper-with-me

홈 › Papers

Papez: Resource-Efficient Speech Separation with Auditory Working Memory

2024-07-01 · Hyunseok Oh, Juheon Yi, Youngki Lee

Transformer-based models recently reached state-of-the-art single-channel speech separation accuracy; However, their extreme computational load makes it difficult to deploy them in resource-constrained mobile or IoT devices. We thus present Papez, a lightweight and computation-efficient single-channel speech separation model. Papez is based on three key techniques. We first replace the inter-chunk Transformer with small-sized auditory working memory. Second, we adaptively prune the input tokens that do not need further processing. Finally, we reduce the number of parameters through the recurrent transformer. Our extensive evaluation shows that Papez achieves the best resource and accuracy tradeoffs with a large margin. We publicly share our source code at \texttt{https://github.com/snuhcs/Papez}

📄 PDF Abstract BibTeX arXiv:2407.00888

Code (1)

snuhcs/papez 공식 구현 pytorch

Tasks

Speech Separation

Methods 이 논문이 사용한 방법론

Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Multi-Head Attention 설명 없음
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…
Position-Wise Feed-Forward Layer 설명 없음

Similar Papers 제목 키워드 기반

An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

2022-12-21 · Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan 외

Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active resear…

Speech Separation

A General Close-loop Predictive Coding Framework for Auditory Working Memory

2025-03-16 · Zhongju Yuan, Geraint Wiggins, Dick Botteldooren

Auditory working memory is essential for various daily activities, such as language acquisition, conversation. It involves the temporary storage and manipulation of information that is no longer present in the environmen…

Language AcquisitionSemantic SimilaritySemantic Textual Similarity

TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion

2024-01-25 · Samuel Pegg, Kai Li, Xiaolin Hu

Audio-visual speech separation has gained significant traction in recent years due to its potential applications in various fields such as speech recognition, diarization, scene analysis and assistive technologies. Desig…

speech-recognitionSpeech RecognitionSpeech Separation

Pac-HuBERT: Self-Supervised Music Source Separation via Primitive Auditory Clustering and Hidden-Unit BERT

2023-04-04 · Ke Chen, Gordon Wichern, François G. Germain, Jonathan Le Roux

In spite of the progress in music source separation research, the small amount of publicly-available clean source data remains a constant limiting factor for performance. Thus, recent advances in self-supervised learning…

ClusteringDecoderMusic Source SeparationSelf-Supervised Learning

Jointly Learning Visual and Auditory Speech Representations from Raw Data

2022-12-12 · Alexandros Haliassos, Pingchuan Ma, Rodrigo Mira, Stavros Petridis 외

We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets…

Audio-Visual Speech RecognitionLipreadingspeech-recognitionSpeech Recognition+1