GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text
Large language models have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing language models cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal large language model that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%-10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.
Code (1)
Tasks
Drug DiscoveryImage CaptioningLanguage ModelingLanguage ModellingLarge Language Modelmolecular representationMolecule CaptioningProperty PredictionText-based de novo Molecule GenerationSimilar Papers 제목 키워드 기반
A quantitative analysis of knowledge-learning preferences in large language models in molecular science
Deep learning has significantly advanced molecular modeling and design, enabling efficient understanding and discovery of novel molecules. In particular, large language models (LLMs) introduce a fresh research paradigm t…
Language ModellingBridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule
Artificial intelligence has demonstrated immense potential in scientific research. Within molecular science, it is revolutionizing the traditional computer-aided paradigm, ushering in a new era of deep learning. With rec…
Drug DiscoveryReasoning-Enhanced Large Language Models for Molecular Property Prediction
Molecular property prediction is crucial for drug discovery and materials science, yet existing approaches suffer from limited interpretability, poor cross-task generalization, and lack of chemical reasoning capabilities…
Molecular Property PredictionReinforcement LearningDrug DiscoveryProtTeX: Structure-In-Context Reasoning and Editing of Proteins with Large Language Models
Large language models have made remarkable progress in the field of molecular science, particularly in understanding and generating functional small molecules. This success is largely attributed to the effectiveness of m…
multimodal generationProtein DesignProtein Function PredictionA Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimization
Large language models (LLMs) are introducing a paradigm shift in molecular discovery by enabling text-guided interaction with chemical spaces through natural language, symbolic notations, with emerging extensions to inco…
Survey