EM-Network: Oracle Guided Self-distillation for Sequence Learning

2023-06-14 · Ji Won Yoon, Sunghwan Ahn, Hyeonseung Lee, Minchan Kim, Seok Min Kim, Nam Soo Kim

We introduce EM-Network, a novel self-distillation approach that effectively leverages target information for supervised sequence-to-sequence (seq2seq) learning. In contrast to conventional methods, it is trained with oracle guidance, which is derived from the target sequence. Since the oracle guidance compactly represents the target-side context that can assist the sequence model in solving the task, the EM-Network achieves a better prediction compared to using only the source input. To allow the sequence model to inherit the promising capability of the EM-Network, we propose a new self-distillation strategy, where the original sequence model can benefit from the knowledge of the EM-Network in a one-stage manner. We conduct comprehensive experiments on two types of seq2seq models: connectionist temporal classification (CTC) for speech recognition and attention-based encoder-decoder (AED) for machine translation. Experimental results demonstrate that the EM-Network significantly advances the current state-of-the-art approaches, improving over the best prior work on speech recognition and establishing state-of-the-art performance on WMT'14 and IWSLT'14.

📄 PDF Abstract BibTeX arXiv:2306.10058

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Machine Translation speech-recognition Speech Recognition

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Seq2Seq Seq2Seq, or Sequence To Sequence, is a model used in sequence prediction tasks, such as language modelling and machine translation. The idea is to use one…

EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillation

2026-03-10 · Jiajun Cao, Xiaoan Zhang, Xiaobao Wei, Liyuqiu Huang 외 arxiv

Vision-Language-Action models have shown great promise for autonomous driving, yet they suffer from degraded perception after unfreezing the visual encoder and struggle with accumulated instability in long-term planning.…

Autonomous Driving

GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

2026-05-28 · Xiaohang Tang, Keyue Jiang, Che Liu, Qifang Zhao 외 arxiv

Reinforcement learning (RL) can be used to improve the policy (denoiser) of diffusion large language models (dLLMs), while being hindered by the intractability of the policy likelihood. A dominant and efficient family of…

Reinforcement Learning

How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models

2026-06-05 · Hongxing Wang, Harenome Razanajato, Zhen Zhang, Yujie Yuan 외 arxiv

Long-context prefill remains expensive because full/GQA layers still score the historical sequence, even in hybrid models with local, sparse, linear, or recurrent components. We study how much dense attention is needed t…

Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillation

2024-07-28 · Heejoon Koo

In this paper, we present NECHO v2, a novel framework designed to enhance the predictive accuracy of multimodal sequential patient diagnoses under uncertain missing visit sequences, a common challenge in real clinical se…

Knowledge Distillation Sequential Diagnosis

AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment

2026-05-18 · Zhenlin Wei, Pu Jian, Yingzhuo Deng, Xiaohan Wang 외 arxiv

The alignment of Large Language Models (LLMs) for complex reasoning heavily relies on Reinforcement Learning with Verifiable Rewards (RLVR). However, standard algorithms like GRPO apply sequence-level rewards uniformly t…