Efficient Training of Transformers for Molecule Property Prediction on Small-scale Datasets
The blood-brain barrier (BBB) serves as a protective barrier that separates the brain from the circulatory system, regulating the passage of substances into the central nervous system. Assessing the BBB permeability of potential drugs is crucial for effective drug targeting. However, traditional experimental methods for measuring BBB permeability are challenging and impractical for large-scale screening. Consequently, there is a need to develop computational approaches to predict BBB permeability. This paper proposes a GPS Transformer architecture augmented with Self Attention, designed to perform well in the low-data regime. The proposed approach achieved a state-of-the-art performance on the BBB permeability prediction task using the BBBP dataset, surpassing existing models. With a ROC-AUC of 78.8%, the approach sets a state-of-the-art by 5.5%. We demonstrate that standard Self Attention coupled with GPS transformer performs better than other variants of attention coupled with GPS Transformer.
Code (0)
등록된 구현이 없습니다.
Tasks
Property PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their…
Molecular Property Predictionmolecular representationPredictionProperty Prediction+1Chemical Language Models for Natural Products: A State-Space Model Approach
Language models are widely used in chemistry for molecular property prediction and small-molecule generation, yet Natural Products (NPs) remain underexplored despite their importance in drug discovery. To address this ga…
Molecular Property PredictionDrug DiscoveryGeometric Transformer for End-to-End Molecule Properties Prediction
Transformers have become methods of choice in many applications thanks to their ability to represent complex interactions between elements. However, extending the Transformer architecture to non-sequential data such as m…
Property PredictionTransferring a molecular foundation model for polymer property predictions
Transformer-based large language models have remarkable potential to accelerate design optimization for applications such as drug development and materials discovery. Self-supervised pretraining of transformer models req…
Data AugmentationTransfer LearningTransformers for molecular property prediction: Domain adaptation efficiently improves performance
Most of the current transformer-based chemical language models are pre-trained on millions to billions of molecules. However, the improvement from such scaling in dataset size is not confidently linked to improved molecu…
Domain AdaptationMolecular Property PredictionProperty Prediction