Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation

2024-03-03 · Heegon Jin, Seonil Son, Jemin Park, Youngseok Kim, Hyungjong Noh, Yeonsoo Lee

The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation. Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De->Dsb and WMT-2014 En->De, respectively, compared to Transformer baselines.

📄 PDF Abstract BibTeX arXiv:2403.01479

Code (1)

ncsoft/Align-to-Distill 공식 구현 pytorch

Tasks

Knowledge Distillation Machine Translation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Multi-Head Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Enhancing Medical Large Vision-Language Models via Alignment Distillation

2025-12-21 · Aofei Chang, Ting Wang, Fenglong Ma arxiv

Medical Large Vision-Language Models (Med-LVLMs) have shown promising results in clinical applications, but often suffer from hallucinated outputs due to misaligned visual understanding. In this work, we identify two fun…

Visual Question Answering Medical Report Generation Representation Learning

Weight-Inherited Distillation for Task-Agnostic BERT Compression

2023-05-16 · Taiqiang Wu, Cheng Hou, Shanshan Lao, Jiayi Li 외

Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the behavior of the teacher model. These meth…

Knowledge Distillation

Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

2025-02-11 · Zhaodong Bing, Linze Li, Jiajun Liang

Knowledge distillation (KD) in transformers often faces challenges due to misalignment in the number of attention heads between teacher and student models. Existing methods either require identical head counts or introdu…

image-classification Image Classification Image Generation Knowledge Distillation+1

Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition

2021-02-28 · Hirofumi Inaguma, Tatsuya Kawahara

This article describes an efficient training method for online streaming attention-based encoder-decoder (AED) automatic speech recognition (ASR) systems. AED models have achieved competitive performance in offline scena…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder Knowledge Distillation+2

SRA: Span Representation Alignment for Large Language Model Distillation

2026-05-02 · Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi, Tung Nguyen 외 arxiv

Cross-Tokenizer Knowledge Distillation (CTKD) enables knowledge transfer between a large language model and a smaller student, even when they employ different tokenizers. While existing approaches mainly focus on token-l…