Transformers are efficient hierarchical chemical graph learners
Transformers, adapted from natural language processing, are emerging as a leading approach for graph representation learning. Contemporary graph transformers often treat nodes or edges as separate tokens. This approach leads to computational challenges for even moderately-sized graphs due to the quadratic scaling of self-attention complexity with token count. In this paper, we introduce SubFormer, a graph transformer that operates on subgraphs that aggregate information by a message-passing mechanism. This approach reduces the number of tokens and enhances learning long-range interactions. We demonstrate SubFormer on benchmarks for predicting molecular properties from chemical structures and show that it is competitive with state-of-the-art graph transformers at a fraction of the computational cost, with training times on the order of minutes on a consumer-grade graphics card. We interpret the attention weights in terms of chemical structures. We show that SubFormer exhibits limited over-smoothing and avoids over-squashing, which is prevalent in traditional graph neural networks.
Code (1)
Tasks
Graph Representation LearningRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
Contemporary graph learning algorithms are not well-defined for large molecules since they do not consider the hierarchical interactions among the atoms, which are essential to determine the molecular properties of macro…
Graph ClassificationGraph LearningGraph RegressionEnhancing Graph Transformers with Hierarchical Distance Structural Encoding
Graph transformers need strong inductive biases to derive meaningful attention scores. Yet, current methods often fall short in capturing longer ranges, hierarchical structures, or community structures, which are common …
Graph ClassificationGraph RegressionNode ClassificationMOL-Mamba: Enhancing Molecular Representation with Structural & Electronic Insights
Molecular representation learning plays a crucial role in various downstream tasks, such as molecular property prediction and drug design. To accurately represent molecules, Graph Neural Networks (GNNs) and Graph Transfo…
Drug DesignMambaMolecular Property Predictionmolecular representation+2BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learning
Graph Transformers have recently attracted attention for molecular property prediction by combining the inductive biases of graph neural networks (GNNs) with the global receptive field of Transformers. However, many exis…
Molecular Property PredictionRepresentation LearningLearning Hierarchical Interaction for Accurate Molecular Property Prediction
Discovering molecules with desirable molecular properties, including ADMET profiles, is of great importance in drug discovery. Existing approaches typically employ deep learning models, such as Graph Neural Networks (GNN…
Drug DiscoveryMolecular Property PredictionPredictionProperty Prediction+1