Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation

2023-05-19 · Kangwook Jang, Sungnyun Kim, Se-Young Yun, Hoirin Kim

Transformer-based speech self-supervised learning (SSL) models, such as HuBERT, show surprising performance in various speech processing tasks. However, huge number of parameters in speech SSL models necessitate the compression to a more compact model for wider usage in academia or small companies. In this study, we suggest to reuse attention maps across the Transformer layers, so as to remove key and query parameters while retaining the number of layers. Furthermore, we propose a novel masking distillation strategy to improve the student model's speech representation quality. We extend the distillation loss to utilize both masked and unmasked speech frames to fully leverage the teacher model's high-quality representation. Our universal compression strategy yields the student model that achieves phoneme error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB benchmark.

📄 PDF Abstract BibTeX arXiv:2305.11685

Code (1)

sungnyun/armhubert 공식 구현 pytorch

Tasks

Self-Supervised Learning

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

Residual Connection 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

ReCycle: Fast and Efficient Long Time Series Forecasting with Residual Cyclic Transformers

2024-05-06 · Arvid Weyrauch, Thomas Steens, Oskar Taubert, Benedikt Hanke 외

Transformers have recently gained prominence in long time series forecasting by elevating accuracies in a variety of use cases. Regrettably, in the race for better predictive performance the overhead of model architectur…

Edge-computing Time Series Time Series Forecasting

Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference

2023-06-04 · Wangchunshu Zhou, Ronan Le Bras, Yejin Choi

Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. C…

Decoder Knowledge Distillation Model Compression Neural Network Compression+2

XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

2021-06-08 · Subhabrata Mukherjee, Ahmed Hassan Awadallah, Jianfeng Gao

While deep and large pre-trained models are the state-of-the-art for various natural language processing tasks, their huge size poses significant challenges for practical uses in resource constrained settings. Recent wor…

Knowledge Distillation NER Question Answering

Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data

2023-07-07 · Qing Xu, Min Wu, XiaoLi Li, Kezhi Mao 외

For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource-limited environments (e.g., smartphones). Moreover, due to the inevitable doma…

Knowledge Distillation Model Compression Time Series Transfer Learning

Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning

2024-08-13 · Shibo Jie, Yehui Tang, Jianyuan Guo, Zhi-Hong Deng 외

Token compression expedites the training and inference of Vision Transformers (ViTs) by reducing the number of the redundant tokens, e.g., pruning inattentive tokens or merging similar tokens. However, when applied to do…

Fine-Grained Image Classification