Machine Translation Verbosity Control for Automatic Dubbing
Automatic dubbing aims at seamlessly replacing the speech in a video document with synthetic speech in a different language. The task implies many challenges, one of which is generating translations that not only convey the original content, but also match the duration of the corresponding utterances. In this paper, we focus on the problem of controlling the verbosity of machine translation output, so that subsequent steps of our automatic dubbing pipeline can generate dubs of better quality. We propose new methods to control the verbosity of MT output and compare them against the state of the art with both intrinsic and extrinsic evaluations. For our experiments we use a public data set to dub English speeches into French, Italian, German and Spanish. Finally, we report extensive subjective tests that measure the impact of MT verbosity control on the final quality of dubbed video clips.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
From Speech-to-Speech Translation to Automatic Dubbing
We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of …
Machine TranslationSpeech-to-Speech Translationtext-to-speechText to Speech+1Isochrony-Aware Neural Machine Translation for Automatic Dubbing
We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structur…
Machine TranslationSentenceTranslationIsometric MT: Neural Machine Translation for Automatic Dubbing
Automatic dubbing (AD) is among the machine translation (MT) use cases where translations should match a given length to allow for synchronicity between source and target speech. For neural MT, generating translations of…
Machine TranslationRe-RankingSelf-LearningTranslationDubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing
We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-…
TranslationThe Two Shades of Dubbing in Neural Machine Translation
Dubbing has two shades; synchronisation constraints are applied only when the actor{'}s mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation r…
Machine TranslationTranslationVocal Bursts Valence Prediction