Average Token Delay: A Latency Metric for Simultaneous Translation
Simultaneous translation is a task in which translation begins before the speaker has finished speaking. In its evaluation, we have to consider the latency of the translation in addition to the quality. The latency is preferably as small as possible for users to comprehend what the speaker says with a small delay. Existing latency metrics focus on when the translation starts but do not consider adequately when the translation ends. This means such metrics do not penalize the latency caused by a long translation output, which actually delays users' comprehension. In this work, we propose a novel latency evaluation metric called Average Token Delay (ATD) that focuses on the end timings of partial translations in simultaneous translation. We discuss the advantage of ATD using simulated examples and also investigate the differences between ATD and Average Lagging with simultaneous translation experiments.
Code (0)
등록된 구현이 없습니다.
Tasks
TranslationSimilar Papers 제목 키워드 기반
Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translation
Simultaneous translation is a task in which the translation begins before the end of an input speech segment. Its evaluation should be conducted based on latency in addition to quality, and for users, the smallest possib…
TranslationA Delay Metric for Video Object Detection: What Average Precision Fails to Tell
Average precision (AP) is a widely used metric to evaluate detection accuracy of image and video object detectors. In this paper, we analyze object detection from videos and point out that AP alone is not sufficient to c…
Objectobject-detectionObject DetectionVideo Object DetectionTextless Streaming Speech-to-Speech Translation using Semantic Speech Tokens
Cascaded speech-to-speech translation systems often suffer from the error accumulation problem and high latency, which is a result of cascaded modules whose inference delays accumulate. In this paper, we propose a transd…
Language ModelingLanguage ModellingMachine TranslationSpeech-to-Speech Translation+3Thinking Slow about Latency Evaluation for Simultaneous Machine Translation
Simultaneous machine translation attempts to translate a source sentence before it is finished being spoken, with applications to translation of spoken language for live streaming and conversation. Since simultaneous sys…
Machine TranslationSentenceTranslationSimultaneous Translation with Flexible Policy via Restricted Imitation Learning
Simultaneous translation is widely useful but remains one of the most difficult tasks in NLP. Previous work either uses fixed-latency policies, or train a complicated two-staged model using reinforcement learning. We pro…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1