Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement

2020-03-29 · Alireza Mohammadshahi, James Henderson

We propose the Recursive Non-autoregressive Graph-to-Graph Transformer architecture (RNGTr) for the iterative refinement of arbitrary graphs through the recursive application of a non-autoregressive Graph-to-Graph Transformer and apply it to syntactic dependency parsing. We demonstrate the power and effectiveness of RNGTr on several dependency corpora, using a refinement model pre-trained with BERT. We also introduce Syntactic Transformer (SynTr), a non-recursive parser similar to our refinement model. RNGTr can improve the accuracy of a variety of initial parsers on 13 languages from the Universal Dependencies Treebanks, English and Chinese Penn Treebanks, and the German CoNLL2009 corpus, even improving over the new state-of-the-art results achieved by SynTr, significantly improving the state-of-the-art for all corpora tested.

📄 PDF Abstract BibTeX arXiv:2003.13118

Code (1)

idiap/g2g-transformer 공식 구현 pytorch

Tasks

Dependency Parsing

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Weight Decay 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation

2022-10-11 · Chenze Shao, Zhengrui Ma, Yang Feng

Non-autoregressive models achieve significant decoding speedup in neural machine translation but lack the ability to capture sequential dependency. Directed Acyclic Transformer (DA-Transformer) was recently proposed to m…

Machine Translation Translation

A Semi-Autoregressive Graph Generative Model for Dependency Graph Parsing

2023-06-21 · Ye Ma, Mingming Sun, Ping Li

Recent years have witnessed the impressive progress in Neural Dependency Parsing. According to the different factorization approaches to the graph joint probabilities, existing parsers can be roughly divided into autoreg…

Dependency Parsing Graph Generation

Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

2026-06-10 · Sanjay Adhikesaven, Haoxiang Sun, Sewon Min arxiv

Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions. These dependencies are recursive: a model may depend on an upstream artifa…

Information Extraction

Graph-to-Graph Transformer for Transition-based Dependency Parsing

2019-11-08 · Findings of the Association for Computational Linguistics 2020 · Alireza Mohammadshahi, James Henderson

We propose the Graph2Graph Transformer architecture for conditioning on and predicting arbitrary graphs, and apply it to the challenging task of transition-based dependency parsing. After proposing two novel Transformer …

Dependency Parsing Structured Prediction Transition-Based Dependency Parsing

Online Bayesian system identification in multivariate autoregressive models via message passing

2025-06-03 · T. N. Nisslbeck, Wouter M. Kouw

We propose a recursive Bayesian estimation procedure for multivariate autoregressive models with exogenous inputs based on message passing in a factor graph. Unlike recursive least-squares, our method produces full poste…