State-of-the-Art Augmented NLP Transformer models for direct and single-step retrosynthesis

2020-03-05 · Igor V. Tetko, Pavel Karpov, Ruud Van Deursen, Guillaume Godin

We investigated the effect of different training scenarios on predicting the (retro)synthesis of chemical compounds using a text-like representation of chemical reactions (SMILES) and Natural Language Processing neural network Transformer architecture. We showed that data augmentation, which is a powerful method used in image processing, eliminated the effect of data memorization by neural networks, and improved their performance for the prediction of new sequences. This effect was observed when augmentation was used simultaneously for input and the target data simultaneously. The top-5 accuracy was 84.8% for the prediction of the largest fragment (thus identifying principal transformation for classical retro-synthesis) for the USPTO-50k test dataset and was achieved by a combination of SMILES augmentation and a beam search algorithm. The same approach provided significantly better results for the prediction of direct reactions from the single-step USPTO-MIT test set. Our model achieved 90.6% top-1 and 96.1% top-5 accuracy for its challenging mixed set and 97% top-5 accuracy for the USPTO-MIT separated set. It also significantly improved results for USPTO-full set single-step retrosynthesis for both top-1 and top-10 accuracies. The appearance frequency of the most abundantly generated SMILES was well correlated with the prediction outcome and can be used as a measure of the quality of reaction prediction.

📄 PDF Abstract BibTeX arXiv:2003.02804

Code (1)

bigchem/synthesis 공식 구현 tf

Tasks

Data Augmentation Memorization Prediction Retrosynthesis Single-step retrosynthesis

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

State-space models can learn in-context by gradient descent

2024-10-15 · Neeraj Mohan Sushma, Yudou Tian, Harshvardhan Mestha, Nicolo Colombo 외

Deep state-space models (Deep SSMs) are becoming popular as effective approaches to model sequence data. They have also been shown to be capable of in-context learning, much like transformers. However, a complete picture…

In-Context Learning regression State Space Models

Forget Attention: Importance-Aware Attention Is All You Need

2026-06-01 · Suhyeong Shin, Yeongwook Yang arxiv

Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling. Transformers see everywhere but cannot prioritize; SSMs know wh…

T6D-Direct: Transformers for Multi-Object 6D Pose Direct Regression

2021-09-22 · Arash Amini, Arul Selvam Periyasamy, Sven Behnke

6D pose estimation is the task of predicting the translation and orientation of objects in a given input image, which is a crucial prerequisite for many robotics and augmented reality applications. Lately, the Transforme…

6D Pose Estimation object-detection Object Detection Pose Estimation+2

MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions

2023-02-23 · Shengkui Zhao, Bin Ma

Transformer based models have provided significant performance improvements in monaural speech separation. However, there is still a performance gap compared to a recent proposed upper bound. The major limitation of the …

Speech Separation

An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition

2021-06-18 · Ruchao Fan, Wei Chu, Peng Chang, Jing Xiao 외

Non-autoregressive mechanisms can significantly decrease inference time for speech transformers, especially when the single step variant is applied. Previous work on CTC alignment-based single step non-autoregressive tra…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Decoder Language Modelling+3