Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata

2024-04-10 · Jinghong Chen, Weizhe Lin, Jingbiao Mei, Bill Byrne

The Directed Acyclic Transformer is a fast non-autoregressive (NAR) model that performs well in Neural Machine Translation. Two issues prevent its application to general Natural Language Generation (NLG) tasks: frequent Out-Of-Vocabulary (OOV) errors and the inability to faithfully generate entity names. We introduce Control-DAG, a constrained decoding algorithm for our Directed Acyclic T5 (DA-T5) model which offers lexical, vocabulary and length control. We show that Control-DAG significantly enhances DA-T5 on the Schema Guided Dialogue and the DART datasets, establishing strong NAR results for Task-Oriented Dialogue and Data-to-Text NLG.

📄 PDF Abstract BibTeX arXiv:2404.06854

Code (1)

erichen0615/controldag 공식 구현 pytorch

Tasks

Machine Translation Text Generation Translation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

DART # 🎯 DART-Math > Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving 📝 Paper@arXiv | 🤗…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adafactor Adafactor is a stochastic optimization method based on Adam that reduces memory usage while retaining the empirical benefits of…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation

2022-10-11 · Chenze Shao, Zhengrui Ma, Yang Feng

Non-autoregressive models achieve significant decoding speedup in neural machine translation but lack the ability to capture sequential dependency. Directed Acyclic Transformer (DA-Transformer) was recently proposed to m…

Machine Translation Translation

A Decoding Algorithm for Length-Control Summarization Based on Directed Acyclic Transformers

2025-02-06 · Chenyang Huang, Hao Zhou, Cameron Jen, Kangjie Zheng 외

Length-control summarization aims to condense long texts into a short one within a certain length limit. Previous approaches often use autoregressive (AR) models and treat the length requirement as a soft constraint, whi…

Directed Acyclic Transformer for Non-Autoregressive Machine Translation

2022-05-16 · Fei Huang, Hao Zhou, Yang Liu, Hang Li 외

Non-autoregressive Transformers (NATs) significantly reduce the decoding latency by generating all tokens in parallel. However, such independent predictions prevent NATs from capturing the dependencies between the tokens…

Knowledge Distillation Machine Translation Translation

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation

2023-04-24 · Fei Huang, Pei Ke, Minlie Huang

Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation. However, in a wider range of text generat…

Machine Translation Text Generation

Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation

2023-03-12 · Zhengrui Ma, Chenze Shao, Shangtong Gui, Min Zhang 외

Non-autoregressive translation (NAT) reduces the decoding latency but suffers from performance degradation due to the multi-modality problem. Recently, the structure of directed acyclic graph has achieved great success i…

Machine Translation Sentence Translation