Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches

2025-01-13 · Gabriel Bianchin de Oliveira, Helio Pedrini, Zanoni Dias

Various approaches utilizing Transformer architectures have achieved state-of-the-art results in Natural Language Processing (NLP). Based on this success, numerous architectures have been proposed for other types of data, such as in biology, particularly for protein sequences. Notably among these are the ESM2 architectures, pre-trained on billions of proteins, which form the basis of various state-of-the-art approaches in the field. However, the ESM2 architectures have a limitation regarding input size, restricting it to 1,022 amino acids, which necessitates the use of preprocessing techniques to handle sequences longer than this limit. In this paper, we present the long and quantized versions of the ESM2 architectures, doubling the input size limit to 2,048 amino acids.

📄 PDF Abstract BibTeX arXiv:2501.07747

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Adam 설명 없음

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

RITA: a Study on Scaling Up Generative Protein Sequence Models

2022-05-11 · Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli 외

In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database. Su…

Prediction Protein Design

Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences

2024-11-06 · Niklas Schmidinger, Lisa Schneckenreiter, Philipp Seidl, Johannes Schimunek 외

Language models for biological and chemical sequences enable crucial applications such as drug discovery, protein engineering, and precision medicine. Currently, these language models are predominantly based on Transform…

Drug Discovery In-Context Learning

Quantitative Molecular Scaling Theory of Protein Amino Acid Sequences, Structure, and Functionality

2016-10-13

Here we review the development of protein scaling theory, starting from backgrounds in mathematics and statistical mechanics, and leading to biomedical applications. Evolution has organized each protein family in differe…

Articles

XAI-Driven Deep Learning for Protein Sequence Functional Group Classification

2025-11-16 · Pratik Chakraborty, Aryan Bhargava arxiv

Proteins perform essential biological functions, and accurate classification of their sequences is critical for understanding structure-function relationships, enzyme mechanisms, and molecular interactions. This study pr…

Protein Structure Prediction in the 3D HP Model Using Deep Reinforcement Learning

2024-12-29 · Giovanny Espitia, Yui Tik Pang, James C. Gumbart

We address protein structure prediction in the 3D Hydrophobic-Polar lattice model through two novel deep learning architectures. For proteins under 36 residues, our hybrid reservoir-based model combines fixed random proj…

Deep Reinforcement Learning Protein Structure Prediction Q-Learning