Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction
Vision-Language Models (VLMs) have shown strong performance in tasks like visual question answering and multimodal text generation, but their effectiveness in scientific domains such as materials science remains limited. While some machine learning methods have addressed specific challenges in this field, there is still a lack of foundation models designed for broad tasks like polymer property prediction using multimodal data. In this work, we present a multimodal polymer dataset to fine-tune VLMs through instruction-tuning pairs and assess the impact of multimodality on prediction performance. Our fine-tuned models, using LoRA, outperform unimodal and baseline approaches, demonstrating the benefits of multimodal learning. Additionally, this approach reduces the need to train separate models for different properties, lowering deployment and maintenance costs.
Code (0)
등록된 구현이 없습니다.
Tasks
Visual Question AnsweringText GenerationSimilar Papers 제목 키워드 기반
MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction
Polymers are high-molecular-weight compounds constructed by the covalent bonding of numerous identical or similar monomers so that their 3D structures are complex yet exhibit unignorable regularity. Typically, the proper…
cross-modal alignmentPredictionProperty PredictionMultimodal machine learning with large language embedding model for polymer property prediction
Contemporary large language models (LLMs), such as GPT-4 and Llama, have harnessed extensive computational power and diverse text corpora to achieve remarkable proficiency in interpreting and generating domain-specific c…
Property PredictionPolyRecommender: A Multimodal Recommendation System for Polymer Discovery
We introduce PolyRecommender, a multimodal discovery framework that integrates chemical language representations from PolyBERT with molecular graph-based representations from a graph encoder. The system first retrieves c…
Multimodal RecommendationLarge language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties
Large language models (LLMs) bear promise as a fast and accurate material modeling paradigm for evaluation, analysis, and design. Their vast number of trainable parameters necessitates a wealth of data to achieve accurac…
Towards Understanding Multimodal Fine-Tuning: Spatial Features
Contemporary Vision-Language Models (VLMs) achieve strong performance on a wide range of tasks by pairing a vision encoder with a pre-trained language model, fine-tuned for visual-text inputs. Yet despite these gains, it…
Visual Grounding