Towards 3D Molecule-Text Interpretation in Language Models
Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties. We release our codes and datasets at https://github.com/lsh0520/3D-MoLM.
Code (1)
Tasks
Instruction FollowingLanguage ModelingLanguage ModellingMolecule CaptioningRetrievalText RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Translation between Molecules and Natural Language
We present $\textbf{MolT5}$ $-$ a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. $\textbf{MolT5}$ allows for new, useful, and challengi…
Drug DiscoveryMolecule CaptioningSelf-Supervised LearningText-based de novo Molecule Generation+1MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Model
Human expertise in chemistry and biomedicine relies on contextual molecular understanding, a capability that large language models (LLMs) can extend through fine-grained alignment between molecular structures and text. R…
cross-modal alignmentLanguage ModelingLanguage ModellingUniMoT: Unified Molecule-Text Language Model with Discrete Token Representation
The remarkable success of Large Language Models (LLMs) across diverse tasks has driven the research community to extend their capabilities to molecular applications. However, most molecular LLMs employ adapter-based arch…
Language ModelingLanguage ModellingQuantizationLarge Language Models are In-Context Molecule Learners
Large Language Models (LLMs) have demonstrated exceptional performance in biochemical tasks, especially the molecule caption translation task, which aims to bridge the gap between molecules and natural language texts. Ho…
Cross-Modal RetrievalIn-Context LearningRe-RankingRetrieval+1Vision Language Model is NOT All You Need: Augmentation Strategies for Molecule Language Models
Recently, there has been a growing interest among researchers in understanding molecules and their textual descriptions through molecule language models (MoLM). However, despite some early promising developments, the adv…
AllDrug DiscoveryLanguage ModelingLanguage Modelling