Dynamically Context-Sensitive Time-Decay Attention for Dialogue Modeling
Spoken language understanding (SLU) is an essential component in conversational systems. Considering that contexts provide informative cues for better understanding, history can be leveraged for contextual SLU. However, most prior work only paid attention to the related content in history utterances and ignored the temporal information. In dialogues, it is intuitive that the most recent utterances are more important than the least recent ones, and time-aware attention should be in a decaying manner. Therefore, this paper allows the model to automatically learn a time-decay attention function where the attentional weights can be dynamically decided based on the content of each role's contexts, which effectively integrates both content-aware and time-aware perspectives and demonstrates remarkable flexibility to complex dialogue contexts. The experiments on the benchmark Dialogue State Tracking Challenge (DSTC4) dataset show that the proposed dynamically context-sensitive time-decay attention mechanisms significantly improve the state-of-the-art model for contextual understanding performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Dialogue State TrackingSpoken Language UnderstandingSimilar Papers 제목 키워드 기반
Learning When Not to Attend Globally
When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs) can learn to dynamically determine whe…
How Time Matters: Learning Time-Decay Attention for Contextual Spoken Language Understanding in Dialogues
Spoken language understanding (SLU) is an essential component in conversational systems. Most SLU components treats each utterance independently, and then the following components aggregate the multi-turn information in …
Dialogue State TrackingImage CaptioningMachine TranslationSentence+3MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a…
Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs
Large Language Models (LLMs) have achieved remarkable performance on single-turn tasks, yet their effectiveness deteriorates in multi-turn conversations. We define this phenomenon as cumulative contextual decay - a progr…
Decay-Function-Free Time-Aware Attention to Context and Speaker Indicator for Spoken Language Understanding
To capture salient contextual information for spoken language understanding (SLU) of a dialogue, we propose time-aware models that automatically learn the latent time-decay function of the history without a manual time-d…
dialog state trackingSpoken Language Understanding