CTRAN: CNN-Transformer-based Network for Natural Language Understanding
Intent-detection and slot-filling are the two main tasks in natural language understanding. In this study, we propose CTRAN, a novel encoder-decoder CNN-Transformer-based architecture for intent-detection and slot-filling. In the encoder, we use BERT, followed by several convolutional layers, and rearrange the output using window feature sequence. We use stacked Transformer encoders after the window feature sequence. For the intent-detection decoder, we utilize self-attention followed by a linear layer. In the slot-filling decoder, we introduce the aligned Transformer decoder, which utilizes a zero diagonal mask, aligning output tags with input tokens. We apply our network on ATIS and SNIPS, and surpass the current state-of-the-art in slot-filling on both datasets. Furthermore, we incorporate the language model as word embeddings, and show that this strategy yields a better result when compared to the language model as an encoder.
Code (1)
Tasks
DecoderIntent DetectionLanguage ModelingLanguage ModellingNatural Language Understandingslot-fillingSlot FillingWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Ranking Creative Language Characteristics in Small Data Scenarios
The ability to rank creative natural language provides an important general tool for downstream language understanding and generation. However, current deep ranking models require substantial amounts of labeled data that…
Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning
This paper presents new hierarchically cascaded transformers that can improve data efficiency through attribute surrogates learning and spectral tokens pooling. Vision transformers have recently been thought of as a prom…
AttributeFew-Shot Image ClassificationFew-Shot LearningSelf-Supervised LearningMulti-Compound Transformer for Accurate Biomedical Image Segmentation
The recent vision transformer(i.e.for image classification) learns non-local attentive interaction of different patch tokens. However, prior arts miss learning the cross-scale dependencies of different pixels, the semant…
image-classificationImage ClassificationImage SegmentationSemantic correspondence+1UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer
Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) N…
DecoderImage SegmentationMedical Image SegmentationSegmentation+2The Volctrans Machine Translation System for WMT20
This paper describes our VolcTrans system on WMT20 shared news translation task. We participated in 8 translation directions. Our basic systems are based on Transformer, with several variants (wider or deeper Transformer…
Machine TranslationSynthetic Data GenerationTranslation