VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation
Existing multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate appropriate translations. We introduce VISA, a new dataset that consists of 40k Japanese-English parallel sentence pairs and corresponding video clips with the following key features: (1) the parallel sentences are subtitles from movies and TV episodes; (2) the source subtitles are ambiguous, which means they have multiple possible translations with different meanings; (3) we divide the dataset into Polysemy and Omission according to the cause of ambiguity. We show that VISA is challenging for the latest MMT system, and we hope that the dataset can facilitate MMT research. The VISA dataset is available at: https://github.com/ku-nlp/VISA.
Code (1)
Tasks
Machine TranslationMultimodal Machine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
LaViSA: A Language and Vision Structural Ambiguity Benchmark
Structural ambiguity arises when a single sentence admits multiple valid interpretations due to its syntactic structure, posing a fundamental challenge for language understanding. Visual scenes serve as useful cues for r…
Video-Helpful Multimodal Machine Translation
Existing multimodal machine translation (MMT) datasets consist of images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity, making visual information ineffective in generating…
Machine TranslationMultimodal Machine TranslationTranslationUnbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasing
Video Scene Graph Generation (VidSGG) aims to capture dynamic relationships among entities by sequentially analyzing video frames and integrating visual and semantic information. However, VidSGG is challenged by signific…
Graph GenerationScene Graph GenerationTripletVideo scene graph generationVISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation
Emotion Recognition in Conversation (ERC) is essential for effective human-machine interaction, aiming to identify speakers' emotional states in multi-turn dialogues. Early text-based methods struggle with complex scenar…
Emotion Recognition in ConversationComputational EfficiencyViSaRL: Visual Reinforcement Learning Guided by Human Saliency
Training robots to perform complex control tasks from high-dimensional pixel input using reinforcement learning (RL) is sample-inefficient, because image observations are comprised primarily of task-irrelevant informatio…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Robot Manipulation