paper-with-me

Papers

Isometric MT: Neural Machine Translation for Automatic Dubbing

2021-12-16 · Surafel M. Lakew, Yogesh Virkar, Prashant Mathur, Marcello Federico

Automatic dubbing (AD) is among the machine translation (MT) use cases where translations should match a given length to allow for synchronicity between source and target speech. For neural MT, generating translations of length close to the source length (e.g. within +-10% in character count), while preserving quality is a challenging task. Controlling MT output length comes at a cost to translation quality, which is usually mitigated with a two step approach of generating N-best hypotheses and then re-ranking based on length and quality. This work introduces a self-learning approach that allows a transformer model to directly learn to generate outputs that closely match the source length, in short Isometric MT. In particular, our approach does not require to generate multiple hypotheses nor any auxiliary ranking function. We report results on four language pairs (English - French, Italian, German, Spanish) with a publicly available benchmark. Automatic and manual evaluations show that our method for Isometric MT outperforms more complex approaches proposed in the literature.

📄 PDF Abstract BibTeX arXiv:2112.08682

Code (0)

등록된 구현이 없습니다.

Tasks

Machine TranslationRe-RankingSelf-LearningTranslation

Methods 이 논문이 사용한 방법론

Self-Learning 설명 없음

Similar Papers 제목 키워드 기반

Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

2022-12-23 · William Brannon, Yogesh Virkar, Brian Thompson

We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-…

Translation

Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning

2024-03-20 · Shivam Ratnakant Mhaskar, Nirmesh J. Shah, Mohammadi Zaki, Ashishkumar P. Gudmalwar 외

Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NM…

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine TranslationNMT+7

From Speech-to-Speech Translation to Automatic Dubbing

2020-01-19 · WS 2020 7 · Marcello Federico, Robert Enyedi, Roberto Barra-Chicote, Ritwik Giri 외

We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of …

Machine TranslationSpeech-to-Speech Translationtext-to-speechText to Speech+1

Isochrony-Aware Neural Machine Translation for Automatic Dubbing

2021-12-16 · Derek Tam, Surafel M. Lakew, Yogesh Virkar, Prashant Mathur 외

We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structur…

Machine TranslationSentenceTranslation

Machine Translation Verbosity Control for Automatic Dubbing

2021-10-08 · Surafel M. Lakew, Marcello Federico, Yue Wang, Cuong Hoang 외

Automatic dubbing aims at seamlessly replacing the speech in a video document with synthetic speech in a different language. The task implies many challenges, one of which is generating translations that not only convey …

Machine TranslationTranslation