Enhancing the Transformer Decoder with Transition-based Syntax

2021-01-29 · Leshem Choshen, Omri Abend

Notwithstanding recent advances, syntactic generalization remains a challenge for text decoders. While some studies showed gains from incorporating source-side symbolic syntactic and semantic structure into text generation Transformers, very little work addressed the decoding of such structure. We propose a general approach for tree decoding using a transition-based approach. Examining the challenging test case of incorporating Universal Dependencies syntax into machine translation, we present substantial improvements on test sets that focus on syntactic generalization, while presenting improved or comparable performance on standard MT benchmarks. Further qualitative analysis addresses cases where syntactic generalization in the vanilla Transformer decoder is inadequate and demonstrates the advantages afforded by integrating syntactic information.

📄 PDF Abstract BibTeX arXiv:2101.12640

Code (1)

borgr/nematus 공식 구현 tf

Tasks

Decoder Machine Translation Text Generation Translation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Attention 설명 없음

Residual Connection 설명 없음

Adam 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Transition-based Neural RST Parsing with Implicit Syntax Features

2018-08-01 · COLING 2018 8 · Nan Yu, Meishan Zhang, Guohong Fu

Syntax has been a useful source of information for statistical RST discourse parsing. Under the neural setting, a common approach integrates syntax by a recursive neural network (RNN), requiring discrete output trees pro…

Discourse Parsing Word Embeddings

StructCoder: Structure-Aware Transformer for Code Generation

2022-06-10 · Sindhu Tipirneni, Ming Zhu, Chandan K. Reddy

There has been a recent surge of interest in automating software engineering tasks using deep learning. This paper addresses the problem of code generation, where the goal is to generate target code given source code in …

Code Generation Code Translation Decoder Text-to-Code Generation

Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

2022-05-12 · NAACL 2022 7 · Ghazi Felhi, Joseph Le Roux, Djamé Seddah

We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituenc…

Decoder Disentanglement Inductive Bias Text Generation

Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs

2026-01-23 · Xinyu Gao, Shaonan Wang, Nai Ding arxiv

Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic str…

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

2026-08-06 · Haris Riaz, Hyungji Kim, Mihai Surdeanu arxiv

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic …