Rewiring the Transformer with Depth-Wise LSTMs

2020-07-13 · Hongfei Xu, Yang song, Qiuhui Liu, Josef van Genabith, Deyi Xiong

Stacking non-linear layers allows deep neural networks to model complicated functions, and including residual connections in Transformer layers is beneficial for convergence and performance. However, residual connections may make the model "forget" distant layers and fail to fuse information from previous layers effectively. Selectively managing the representation aggregation of Transformer layers may lead to better performance. In this paper, we present a Transformer with depth-wise LSTMs connecting cascading Transformer layers and sub-layers. We show that layer normalization and feed-forward computation within a Transformer layer can be absorbed into depth-wise LSTMs connecting pure Transformer attention layers. Our experiments with the 6-layer Transformer show significant BLEU improvements in both WMT 14 English-German / French tasks and the OPUS-100 many-to-many multilingual NMT task, and our deep Transformer experiments demonstrate the effectiveness of depth-wise LSTM on the convergence and performance of deep Transformers.

📄 PDF Abstract BibTeX arXiv:2007.06257

Code (0)

등록된 구현이 없습니다.

Tasks

NMTTime Series Analysis

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

Structural Invariance Matters: Rethinking Graph Rewiring through Graph Metrics

2025-10-23 · Alexandre Benoit, Catherine Aitken, Yu He arxiv

Graph rewiring has emerged as a key technique to alleviate over-squashing in Graph Neural Networks (GNNs) and Graph Transformers by modifying the graph topology to improve information flow. While effective, rewiring inhe…

Node Classification

Over-Squashing in GNNs and Causal Inference of Rewiring Strategies

2025-08-12 · Danial Saber, Amirali Salehi-Abari arxiv

Graph neural networks (GNNs) have exhibited state-of-the-art performance across wide-range of domains such as recommender systems, material design, and drug repurposing. Yet message-passing GNNs suffer from over-squashin…

Graph Classification Node Classification Causal Inference

Edge Rewiring Goes Neural: Boosting Network Resilience without Rich Features

2021-10-18 · Shanchao Yang, Kaili Ma, Baoxiang Wang, Tianshu Yu 외

Improving the resilience of a network is a fundamental problem in network science, which protects the underlying system from natural disasters and malicious attacks. This is traditionally achieved via successive degree-p…

Graph Neural Network reinforcement-learning Reinforcement Learning (RL)

Graph Cascades: Contagion-Based Mesoscopic Rewiring for Structure-Aware Graph Machine Learning

2026-06-03 · Meher Chaitanya, My Le, Luana Ruiz arxiv

We introduce Graph Cascades, a mesoscopic rewiring strategy for Graph Neural Networks (GNNs) and Graph Transformers (GTs) that captures intermediate-scale graph structure beyond purely local edges or fully global attenti…

Nested LSTMs

2018-01-31 · Joel Ruben Antony Moniz, David Krueger

We propose Nested LSTMs (NLSTM), a novel RNN architecture with multiple levels of memory. Nested LSTMs add depth to LSTMs via nesting as opposed to stacking. The value of a memory cell in an NLSTM is computed by an LSTM …

Language Modeling Language Modelling