Enhanced Transformer Architecture for Natural Language Processing

2023-10-17 · Woohyeon Moon, TaeYoung Kim, Bumgeun Park, Dongsoo Har

Transformer is a state-of-the-art model in the field of natural language processing (NLP). Current NLP models primarily increase the number of transformers to improve processing performance. However, this technique requires a lot of training resources such as computing capacity. In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset. As a result, the Enhanced Transformer achieves 202.96% higher BLEU score as compared to the original transformer with the translation dataset.

📄 PDF Abstract BibTeX arXiv:2310.10930

Code (0)

등록된 구현이 없습니다.

Tasks

Translation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Residual Connection 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Applying Recent Innovations from NLP to MOOC Student Course Trajectory Modeling

2020-01-23 · Clarence Chen, Zachary Pardos

This paper presents several strategies that can improve neural network-based predictive methods for MOOC student course trajectory modeling, applying multiple ideas previously applied to tackle NLP (Natural Language Proc…

Trajectory Modeling

The Solution for the AIGC Inference Performance Optimization Competition

2024-07-06 · Sishun Pan, Haonan Xu, Zhonghua Wan, Yang Yang

In recent years, the rapid advancement of large-scale pre-trained language models based on transformer architectures has revolutionized natural language processing tasks. Among these, ChatGPT has gained widespread popula…

Computational Efficiency GPU Marketing

Boosting Objective Scores of a Speech Enhancement Model by MetricGAN Post-processing

2020-06-18 · Szu-Wei Fu, Chien-Feng Liao, Tsun-An Hsieh, Kuo-Hsuan Hung 외

The Transformer architecture has demonstrated a superior ability compared to recurrent neural networks in many different natural language processing applications. Therefore, our study applies a modified Transformer in a …

Speech Enhancement

Advancements in Natural Language Processing: Exploring Transformer-Based Architectures for Text Understanding

2025-03-26 · Tianhao Wu, Yu Wang, Ngoc Quach

Natural Language Processing (NLP) has witnessed a transformative leap with the advent of transformer-based architectures, which have significantly enhanced the ability of machines to understand and generate human-like te…

Model Selection

A Cognitive Study on Semantic Similarity Analysis of Large Corpora: A Transformer-based Approach

2022-07-24 · Praneeth Nemani, Satyanarayana Vollala

Semantic similarity analysis and modeling is a fundamentally acclaimed task in many pioneering applications of natural language processing today. Owing to the sensation of sequential pattern recognition, many neural netw…