Variational Inference-Based Dropout in Recurrent Neural Networks for Slot Filling in Spoken Language Understanding
This paper proposes to generalize the variational recurrent neural network (RNN) with variational inference (VI)-based dropout regularization employed for the long short-term memory (LSTM) cells to more advanced RNN architectures like gated recurrent unit (GRU) and bi-directional LSTM/GRU. The new variational RNNs are employed for slot filling, which is an intriguing but challenging task in spoken language understanding. The experiments on the ATIS dataset suggest that the variational RNNs with the VI-based dropout regularization can significantly improve the naive dropout regularization RNNs-based baseline systems in terms of F-measure. Particularly, the variational RNN with bi-directional LSTM/GRU obtains the best F-measure score.
Code (0)
등록된 구현이 없습니다.
Tasks
slot-fillingSlot FillingSpoken Language UnderstandingVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Neuralizing Regular Expressions for Slot Filling
Neural models and symbolic rules such as regular expressions have their respective merits and weaknesses. In this paper, we study the integration of the two approaches for the slot filling task by converting regular expr…
slot-fillingSlot FillingRecurrent Neural Networks with Pre-trained Language Model Embedding for Slot Filling Task
In recent years, Recurrent Neural Networks (RNNs) based models have been applied to the Slot Filling problem of Spoken Language Understanding and achieved the state-of-the-art performances. In this paper, we investigate …
Language ModelingLanguage Modellingslot-fillingSlot Filling+1A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recu…
Bayesian InferenceDeep LearningLanguage ModellingSentiment Analysis+1Variational Adaptive Noise and Dropout towards Stable Recurrent Neural Networks
This paper proposes a novel stable learning theory for recurrent neural networks (RNNs), so-called variational adaptive noise and dropout (VAND). As stabilizing factors for RNNs, noise and dropout on the internal state o…
Imitation LearningLearning TheoryVariational InferenceAttention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
Attention-based encoder-decoder neural network models have recently shown promising results in machine translation and speech recognition. In this work, we propose an attention-based neural network model for joint intent…
Decoderintent-classificationIntent ClassificationIntent Detection+2