TPDM: Selectively Removing Positional Information for Zero-shot Translation via Token-Level Position Disentangle Module
Due to Multilingual Neural Machine Translation's (MNMT) capability of zero-shot translation, many works have been carried out to fully exploit the potential of MNMT in zero-shot translation. It is often hypothesized that positional information may hinder the MNMT from outputting a robust encoded representation for decoding. However, previous approaches treat all the positional information equally and thus are unable to selectively remove certain positional information. In sharp contrast, this paper investigates how to learn to selectively preserve useful positional information. We describe the specific mechanism of positional information influencing MNMT from the perspective of linguistics at the token level. We design a token-level position disentangle module (TPDM) framework to disentangle positional information at the token level based on the explanation. Our experiments demonstrate that our framework improves zero-shot translation by a large margin while reducing the performance loss in the supervised direction compared to previous works.
Code (0)
등록된 구현이 없습니다.
Tasks
PositionTranslationSimilar Papers 제목 키워드 기반
Deep Compositional Phase Diffusion for Long Motion Sequence Generation
Recent research on motion generation has shown significant progress in generating semantically aligned motion with singular semantics. However, when employing these models to create composite sequences containing multipl…
Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation
Diffusion and flow models have achieved remarkable successes in various applications such as text-to-image generation. However, these models typically rely on the same predetermined denoising schedules during inference f…
DenoisingImage GenerationText to Image GenerationText-to-Image GenerationMechanism of Task-oriented Information Removal in In-context Learning
In-context Learning (ICL) is an emerging few-shot learning paradigm based on modern Language Models (LMs), yet its inner mechanism remains unclear. In this paper, we investigate the mechanism through a novel perspective …
Few-Shot LearningThe Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models
Chain-of-thought (CoT) prompting is necessary for arithmetic in small language models, yet shuffling its steps preserves most performance. What does CoT contribute if not logical sequencing? In three 1-3B instruction-tun…
Improving Zero-Shot Translation by Disentangling Positional Information
Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation. Despite being conceptually attractive, it often suffers from…
Machine TranslationTranslation