GTAE: Graph-Transformer based Auto-Encoders for Linguistic-Constrained Text Style Transfer

2021-02-01 · Yukai Shi, Sen Zhang, Chenxing Zhou, Xiaodan Liang, Xiaojun Yang, Liang Lin

Non-parallel text style transfer has attracted increasing research interests in recent years. Despite successes in transferring the style based on the encoder-decoder framework, current approaches still lack the ability to preserve the content and even logic of original sentences, mainly due to the large unconstrained model space or too simplified assumptions on latent embedding space. Since language itself is an intelligent product of humans with certain grammars and has a limited rule-based model space by its nature, relieving this problem requires reconciling the model capacity of deep neural networks with the intrinsic model constraints from human linguistic rules. To this end, we propose a method called Graph Transformer based Auto Encoder (GTAE), which models a sentence as a linguistic graph and performs feature extraction and style transfer at the graph level, to maximally retain the content and the linguistic structure of original sentences. Quantitative experiment results on three non-parallel text style transfer tasks show that our model outperforms state-of-the-art methods in content preservation, while achieving comparable performance on transfer accuracy and sentence naturalness.

📄 PDF Abstract BibTeX arXiv:2102.00769

Code (0)

등록된 구현이 없습니다.

Tasks

DecoderSentenceStyle Transfer Text Style Transfer

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

LapEigen 설명 없음

Laplacian PE Laplacian eigenvectors represent a natural generalization of the…

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Mechanistic Interpretability of ASR models using Sparse Autoencoders

2026-05-12 · Dan Pluth, Zachary Nicholas Houghton, Yu Zhou, Vijay K. Gurbani arxiv

Understanding the internal machinations of deep Transformer-based NLP models is more crucial than ever as these models see widespread use in various domains that affect the public at large, such as industry, academia, fi…

Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases

2022-07-26 · Sławomir Dadas

Sentence embeddings are commonly used in text clustering and semantic retrieval tasks. State-of-the-art sentence representation methods are based on artificial neural networks fine-tuned on large collections of manually …

Language Modeling Language Modelling Retrieval Semantic Retrieval+3

To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph

2021-01-16 · Sufeng Duan, Hai Zhao

In this paper, we propose an explanation of representation for self-attention network (SAN) based neural sequence encoders, which regards the information captured by the model and the encoding of the model as graph struc…

Machine Translation Sentence Translation

Graph Masked Autoencoders with Transformers

2022-02-17 · Sixiao Zhang, Hongxu Chen, Haoran Yang, Xiangguo Sun 외

Recently, transformers have shown promising performance in learning graph representations. However, there are still some challenges when applying transformers to real-world scenarios due to the fact that deep transformer…

Decoder Graph Classification Node Classification

Transformers as Graph-to-Graph Models

2023-10-27 · James Henderson, Alireza Mohammadshahi, Andrei C. Coman, Lesly Miculicich

We argue that Transformers are essentially graph-to-graph models, with sequences just being a special case. Attention weights are functionally equivalent to graph edges. Our Graph-to-Graph Transformer architecture makes …