Transformer with Tree-order Encoding for Neural Program Generation

2022-05-30 · Klaudia-Doris Thellmann, Bernhard Stadler, Ricardo Usbeck, Jens Lehmann

While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical information of the underlying programming language syntax has proven to be effective for code generation. Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended the encoding scheme to allow the attention mechanism to also attend over hierarchical positions in the input. Furthermore, we have realized a decoder based on a restrictive grammar graph model to improve the generation accuracy and ensure the well-formedness of the generated code. While we did not surpass the state of the art, our findings suggest that employing a tree-based positional encoding in combination with a shared natural-language subword vocabulary improves generation performance over sequential positional encodings.

📄 PDF Abstract BibTeX arXiv:2206.13354

Code (1)

smartdataanalytics/codecai 공식 구현 pytorch

Tasks

Code Generation Decoder Semantic Parsing

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Adam 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Novel positional encodings to enable tree-structured transformers

2018-09-27 · Vighnesh Leonardo Shiv, Chris Quirk

With interest in program synthesis and similarly ﬂavored problems rapidly increasing, neural models optimized for tree-domain problems are of great value. In the sequence domain, transformers can learn relationships acro…

Program Synthesis Semantic Parsing Translation

Novel positional encodings to enable tree-based transformers

2019-12-01 · NeurIPS 2019 12 · Vighnesh Shiv, Chris Quirk

Neural models optimized for tree-based problems are of great value in tasks like SQL query extraction and program synthesis. On sequence-structured data, transformers have been shown to learn relationships across arbitra…

Program Synthesis Semantic Parsing Translation

AST-Transformer: Encoding Abstract Syntax Trees Efficiently for Code Summarization

2021-12-02 · Ze Tang, Chuanyi Li, Jidong Ge, Xiaoyu Shen 외

Code summarization aims to generate brief natural language descriptions for source code. As source code is highly structured and follows strict programming language grammars, its Abstract Syntax Tree (AST) is often lever…

Code Summarization

Tree Decomposition Attention for AMR-to-Text Generation

2021-08-27 · Lisa Jin, Daniel Gildea

Text generation from AMR requires mapping a semantic graph to a string that it annotates. Transformer-based graph encoders, however, poorly capture vertex dependencies that may benefit sequence prediction. To impose orde…

AMR-to-Text Generation Text Generation

An empirical study on the limitation of Transformers in program trace generation

2025-09-29 · Simeng Sun arxiv

We study Transformers on the task \emph{program trace generation} (PTG), where models produce step-by-step execution traces for synthetic programs. Unlike existing algorithmic problems, PTG externalizes reasoning through…