Modular and Parameter-Efficient Multimodal Fusion with Prompting
Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning. In this paper, we propose to use prompt vectors to align the modalities. Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings. We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Modular and Parameter-Efficient Multimodal Fusion with Prompting
Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than fine-tuning. In thi…
Conditional Prompt Tuning for Multimodal Fusion
We show that the representation of one modality can effectively guide the prompting of another modality for parameter-efficient multimodal fusion. Specifically, we first encode one modality and use its representation as …
Multimodal Modular Chain of Thoughts in Energy Performance Certificate Assessment
Accurate evaluation of building energy performance remains challenging in regions where scalable Energy Performance Certificate (EPC) assessments are unavailable. This paper presents a cost-efficient framework that lever…
Efficient Multimodal Fusion via Interactive Prompting
Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era. Following this trend, the size of multi-modal learning models constantly increases, leading to an…
ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way
We introduce ByDeWay, a training-free framework designed to enhance the performance of Multimodal Large Language Models (MLLMs). ByDeWay uses a novel prompting strategy called Layered-Depth-Based Prompting (LDP), which i…
Depth EstimationHallucinationLanguage ModelingLanguage Modelling+2