홈 › Papers

ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning

2025-07-09 · Yichen Lu, Wei Dai, Jiaen Liu, Ching Wing Kwok, Zongheng Wu, Xudong Xiao, Ao Sun, Sheng Fu, Jianyuan Zhan, Yian Wang, Takatomo Saito, Sicheng Lai arxiv

LLM-based translation agents have achieved highly human-like translation results and are capable of handling longer and more complex contexts with greater efficiency. However, they are typically limited to text-only inputs. In this paper, we introduce ViDove, a translation agent system designed for multimodal input. Inspired by the workflow of human translators, ViDove leverages visual and contextual background information to enhance the translation process. Additionally, we integrate a multimodal memory system and long-short term memory modules enriched with domain-specific knowledge, enabling the agent to perform more accurately and adaptively in real-world scenarios. As a result, ViDove achieves significantly higher translation quality in both subtitle generation and general translation tasks, with a 28% improvement in BLEU scores and a 15% improvement in SubER compared to previous state-of-the-art baselines. Moreover, we introduce DoveBench, a new benchmark for long-form automatic video subtitling and translation, featuring 17 hours of high-quality, human-annotated data. Our code is available here: https://github.com/pigeonai-org/ViDove

📄 PDF Abstract BibTeX arXiv:2507.07306

Code (0)

등록된 구현이 없습니다.

MSCTD: A Multimodal Sentiment Chat Translation Dataset

2022-02-28 · ACL 2022 5 · Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen 외

Multimodal machine translation and textual chat translation have received considerable attention in recent years. Although the conversation in its natural form is usually multimodal, there still lacks work on multimodal …

Machine Translation Multimodal Machine Translation Sentiment Analysis Translation

LIUM-CVC Submissions for WMT17 Multimodal Translation Task

2017-07-14 · WS 2017 9 · Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes García-Martínez 외

This paper describes the monomodal and multimodal Neural Machine Translation systems developed by LIUM and CVC for WMT17 Shared Task on Multimodal Translation. We mainly explored two multimodal architectures where either…

Machine Translation Translation

CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation

2025-05-30 · Emilio Villa-Cueva, Sholpan Bolatzhanova, Diana Turmakhan, Kareem Elzeky 외

Cultural content poses challenges for machine translation systems due to the differences in conceptualizations between cultures, where language alone may fail to convey sufficient context to capture region-specific meani…

Benchmarking Machine Translation Multimodal Machine Translation Translation

Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

2021-02-22 · EACL 2021 2 · Julia Ive, Andy Mingren Li, Yishu Miao, Ozan Caglayan 외

This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual…

Machine Translation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages

2023-08-30 · Baban Gain, Dibyanayan Bandyopadhyay, Samrat Mukherjee, Chandranath Adak 외

The study investigates the effectiveness of utilizing multimodal information in Neural Machine Translation (NMT). While prior research focused on using multimodal data in low-resource scenarios, this study examines how i…

Machine Translation NMT Translation