An Introductory Survey on Attention Mechanisms in NLP Problems
First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Dependency ParsingGeneral ClassificationMachine TranslationQuestion AnsweringSentiment AnalysisSentiment ClassificationSurveyText SummarizationTranslationSimilar Papers 제목 키워드 기반
a survey on GPT-3
This paper provides an introductory survey to GPT-3. We cover some of the historical development behind this technology, some of the key features of GPT-3, and discuss the machine learning model and the datasets used. We…
HallucinationSurveyPractical Framework for Problem-Based Learning in an Introductory Circuit Analysis Course
Introductory courses on electric circuits at undergraduate level are usually presented in quite abstract terms, with questions and problems quite far from practical problems. This causes the students have difficulties to…
SchedulingA General Survey on Attention Mechanisms in Deep Learning
Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed…
Deep LearningSurveyAn Introductory Guide to Fano's Inequality with Applications in Statistical Estimation
Information theory plays an indispensable role in the development of algorithm-independent impossibility results, both for communication problems and for seemingly distinct areas such as statistics and machine learning. …
Density EstimationModel SelectionregressionOnline Fair Division: A Survey
We survey a burgeoning and promising new research area that considers the online nature of many practical fair division problems. We identify wide variety of such online fair division problems, as well as discuss new mec…
Survey