Modeling Protein Using Large-scale Pretrain Language Model
Protein is linked to almost every life process. Therefore, analyzing the biological structure and property of protein sequences is critical to the exploration of life, as well as disease detection and drug discovery. Traditional protein analysis methods tend to be labor-intensive and time-consuming. The emergence of deep learning models makes modeling data patterns in large quantities of data possible. Interdisciplinary researchers have begun to leverage deep learning methods to model large biological datasets, e.g. using long short-term memory and convolutional neural network for protein sequence classification. After millions of years of evolution, evolutionary information is encoded in protein sequences. Inspired by the similarity between natural language and protein sequences, we use large-scale language models to model evolutionary-scale protein sequences, encoding protein biology information in representation. Significant improvements are observed in both token-level and sequence-level tasks, demonstrating that our large-scale model can accurately capture evolution information from pretraining on evolutionary-scale individual sequences. Our code and model are available at https://github.com/THUDM/ProteinLM.
Code (1)
Tasks
Drug DiscoveryLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Protap: A Benchmark for Protein Modeling on Realistic Downstream Applications
Recently, extensive deep learning architectures and pretraining strategies have been explored to support downstream protein applications. Additionally, domain-specific models incorporating biological knowledge have been …
Endowing Protein Language Models with Structural Knowledge
Understanding the relationships between protein sequence, structure and function is a long-standing biological challenge with manifold implications from drug design to our understanding of evolution. Recently, protein la…
Drug DesignLanguage ModelingLanguage ModellingMasked Language Modeling+2From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning?
Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures. In parallel, reinforcement learning (RL) has broadened exploration and enabled precis…
Protein Language ModelReinforcement LearningProtein DesignProteinBERT: a universal deep-learning model of protein sequence and function
Self-supervised deep language modeling has shown unprecedented success across natural language tasks, and has recently been repurposed to biological sequences. However, existing models and pretraining methods are designe…
Language ModelingLanguage ModellingProtein Secondary Structure PredictionProtein Structure PredictionCoPeP: Benchmarking Continual Pretraining for Protein Language Models
Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeuti…
Continual PretrainingContinual LearningDrug Discovery