Curriculum Script Distillation for Multilingual Visual Question Answering
Pre-trained models with dual and cross encoders have shown remarkable success in propelling the landscape of several tasks in vision and language in Visual Question Answering (VQA). However, since they are limited by the requirements of gold annotated data, most of these advancements do not see the light of day in other languages beyond English. We aim to address this problem by introducing a curriculum based on the source and target language translations to finetune the pre-trained models for the downstream task. Experimental results demonstrate that script plays a vital role in the performance of these models. Specifically, we show that target languages that share the same script perform better (~6%) than other languages and mixed-script code-switched languages perform better than their counterparts (~5-12%).
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation
Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the reso…
Knowledge DistillationQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)On the Design Fundamentals of Pixel Text Representation Learning
Text-rich visual inputs require models that can read, retrieve, and compress language directly in pixel space, yet existing pixel-text encoders struggle with fixed resolution pretraining, visual shortcut learning, weak v…
Representation LearningVisual GroundingNeural Event Extraction from Movies Description
We present a novel approach for event extraction and abstraction from movie descriptions. Our event frame consists of {``}who{''}, {``}did what{''} {``}to whom{''}, {``}where{''}, and {``}when{''}. We formulate our probl…
Event ExtractionMachine TranslationQuestion AnsweringStory Completion+1Multilingual Image Corpus – Towards a Multimodal and Multilingual Dataset
One of the processing tasks for large multimodal data streams is automatic image description (image classification, object segmentation and classification). Although the number and the diversity of image datasets is cons…
Caption Generationimage-classificationImage ClassificationImage Description+7Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
Self-Distillation Policy Optimization (SDPO) provides dense token-level credit assignment for reinforcement learning with large language models by leveraging the model's own feedback-conditioned predictions as a self-tea…
Reinforcement Learning