Encoder-decoder with Focus-mechanism for Sequence Labelling Based Spoken Language Understanding
This paper investigates the framework of encoder-decoder with attention for sequence labelling based spoken language understanding. We introduce Bidirectional Long Short Term Memory - Long Short Term Memory networks (BLSTM-LSTM) as the encoder-decoder model to fully utilize the power of deep learning. In the sequence labelling task, the input and output sequences are aligned word by word, while the attention mechanism cannot provide the exact alignment. To address this limitation, we propose a novel focus mechanism for encoder-decoder framework. Experiments on the standard ATIS dataset showed that BLSTM-LSTM with focus mechanism defined the new state-of-the-art by outperforming standard BLSTM and attention based encoder-decoder. Further experiments also show that the proposed model is more robust to speech recognition errors.
Code (0)
등록된 구현이 없습니다.
Tasks
Decoderspeech-recognitionSpeech RecognitionSpoken Language UnderstandingSimilar Papers 제목 키워드 기반
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling
We propose a novel deep architecture, SegNet, for semantic pixel wise image labelling. SegNet has several attractive properties; (i) it only requires forward evaluation of a fully learnt function to obtain smooth label p…
DecoderObject CategorizationDistillation of encoder-decoder transformers for sequence labelling
Driven by encouraging results on a wide range of tasks, the field of NLP is experiencing an accelerated race to develop bigger language models. This race for bigger models has also underscored the need to continue the pu…
DecoderFew-Shot LearningHallucinationHybrid Neural Models For Sequence Modelling: The Best Of Three Worlds
We propose a neural architecture with the main characteristics of the most successful neural models of the last years: bidirectional RNNs, encoder-decoder, and the Transformer model. Evaluation on three sequence labellin…
DecoderUnderstanding How Encoder-Decoder Architectures Attend
Encoder-decoder networks with attention have proven to be a powerful way to solve many sequence-to-sequence tasks. In these networks, attention aligns encoder and decoder states and is often used for visualizing network …
DecoderAttention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions
This work proposes an extensive analysis of the Transformer architecture in the Neural Machine Translation (NMT) setting. Focusing on the encoder-decoder attention mechanism, we prove that attention weights systematicall…
DecoderMachine TranslationNMTTranslation+1