Efficient Speech Translation through Model Compression and Knowledge Distillation
Efficient deployment of large audio-language models for speech translation remains challenging due to their significant computational requirements. In this paper, we address this challenge through our system submissions to the "Model Compression" track at the International Conference on Spoken Language Translation (IWSLT 2025). We experiment with a combination of approaches including iterative layer pruning based on layer importance evaluation, low-rank adaptation with 4-bit quantization (QLoRA), and knowledge distillation. In our experiments, we use Qwen2-Audio-7B-Instruct for speech translation into German and Chinese. Our pruned (student) models achieve up to a 50% reduction in both model parameters and storage footprint, while retaining 97-100% of the translation quality of the in-domain (teacher) models.
Code (1)
Tasks
Knowledge DistillationModel CompressionQuantizationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders
Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new…
Contrastive LearningKnowledge DistillationModel Compressionspeech-recognition+4End-to-End Speech Translation with Knowledge Distillation
End-to-end speech translation (ST), which directly translates from source language speech into target language text, has attracted intensive attentions in recent years. Compared to conventional pipeline systems, end-to-e…
Knowledge Distillationspeech-recognitionSpeech RecognitionTranslationNovel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation
Recent advancements in speech-to-text translation have led to the development of multilingual models capable of handling multiple language pairs simultaneously. However, these unified models often suffer from large param…
Speech-to-Text TranslationKnowledge DistillationModel CompressionOn Knowledge Distillation for Direct Speech Translation
Direct speech translation (ST) has shown to be a complex task requiring knowledge transfer from its sub-tasks: automatic speech recognition (ASR) and machine translation (MT). For MT, one of the most promising techniques…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Knowledge DistillationMachine Translation+4SimulSpeech: End-to-End Simultaneous Speech to Text Translation
In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. SimulSpeech consists of a speech enco…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderKnowledge Distillation+9