Injecting linguistic knowledge into BERT for Dialogue State Tracking
Dialogue State Tracking (DST) models often employ intricate neural network architectures, necessitating substantial training data, and their inference process lacks transparency. This paper proposes a method that extracts linguistic knowledge via an unsupervised framework and subsequently utilizes this knowledge to augment BERT's performance and interpretability in DST tasks. The knowledge extraction procedure is computationally economical and does not require annotations or additional training data. The injection of the extracted knowledge can be achieved by the addition of simple neural modules. We employ the Convex Polytopic Model (CPM) as a feature extraction tool for DST tasks and illustrate that the acquired features correlate with syntactic and semantic patterns in the dialogues. This correlation facilitates a comprehensive understanding of the linguistic features influencing the DST model's decision-making process. We benchmark this framework on various DST tasks and observe a notable improvement in accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingDialogue State TrackingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method
Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words - either behind masks or in the next sen…
Semantic SimilaritySemantic Textual SimilaritySentenceUnsupervised Pre-training+1K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when mul…
Dependency ParsingEntity TypingQuestion AnsweringRelation Classification+1GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method
Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words – either through masking or next sentenc…
Semantic SimilaritySemantic Textual SimilaritySentenceUnsupervised Pre-training+1Knowledge Augmented BERT Mutual Network in Multi-turn Spoken Dialogues
Modern spoken language understanding (SLU) systems rely on sophisticated semantic notions revealed in single utterances to detect intents and slots. However, they lack the capability of modeling multi-turn dynamics withi…
Spoken Language UnderstandingCausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision
Recent work has shown success in incorporating pre-trained models like BERT to improve NLP systems. However, existing pre-trained models lack of causal knowledge which prevents today's NLP systems from thinking like huma…
Causal Inference