Diversifying Joint Vision-Language Tokenization Learning
Building joint representations across images and text is an essential step for tasks such as Visual Question Answering and Video Question Answering. In this work, we find that the representations must not only jointly capture features from both modalities but should also be diverse for better generalization performance. To this end, we propose joint vision-language representation learning by diversifying the tokenization learning process, enabling tokens that are sufficiently disentangled from each other to be learned from both modalities. We observe that our approach outperforms the baseline models in a majority of settings and is competitive with state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringRepresentation LearningVideo Question AnsweringVisual Question AnsweringSimilar Papers 제목 키워드 기반
What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?
Tokenization is a fundamental component of language modeling pipelines. Despite its importance, it is often fixed, even though it significantly impacts model performance across languages. In this work, we analyze what to…
Computational EfficiencyJoint Optimization of Tokenization and Downstream Model
Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies imply that the appropriate tokenization …
Machine Translationmodeltext-classificationText Classification+1ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
Action tokenizers play a central role in autoregressive vision-language-action (VLA) models, determining both the targets for policy training and the executable commands recovered from predicted tokens. Their fidelity is…
Representation LearningMaster Thesis: Neural Sign Language Translation by Learning Tokenization
In this thesis, we propose a multitask learning based method to improve Neural Sign Language Translation (NSLT) consisting of two parts, a tokenization layer and Neural Machine Translation (NMT). The tokenization part fo…
Domain AdaptationMachine TranslationNMTSign Language Translation+3End-to-End Vision Tokenizer Tuning
Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual ques…
Image GenerationQuestion AnsweringVisual Question Answering