Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning
In this paper, we present our solution for SMART-101 Challenge of CVPR Multi-modal Algorithmic Reasoning Task 2024. Unlike traditional visual questions and answer tasks, this challenge evaluates abstraction, deduction and generalization ability of neural network in solving visuo-linguistic puzzles designed for specially children in the 6-8 age group. Our model is based on two pre-trained models, dedicated to extract features from text and image respectively. To integrate the features from different modalities, we employed a fusion layer with attention mechanism. We explored different text and image pre-trained models, and fine-tune the integrated classifier on the SMART-101 dataset. Experiment results show that under the data splitting style of puzzle split, our proposed integrated classifier achieves superior performance, verifying the effectiveness of multi-modal pre-trained representations.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
Multimodal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining da…
Generating Multimodal Images with GAN: Integrating Text, Image, and Style
In the field of computer vision, multimodal image generation has become a research hotspot, especially the task of integrating text, image, and style. In this study, we propose a multimodal image generation method based …
Image GenerationJointly Training Large Autoregressive Multimodal Models
In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable o…
Image GenerationZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation
Image-grounded dialogue systems benefit greatly from integrating visual information, resulting in high-quality response generation. However, current models struggle to effectively utilize such information in zero-resourc…
Dialogue GenerationResponse GenerationInvestigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Models
Omnimodal Large Language Models (OLLMs) have shown significant progress in integrating vision and text, but still struggle with integrating vision and audio, often exhibiting suboptimal performance when processing audio …
Knowledge DistillationSelf-Knowledge Distillation