Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?
Emphasis is a crucial component in human communication, which indicates the speaker's intention and implication beyond pure text in dialogue. While Large Language Models (LLMs) have revolutionized natural language processing, their ability to understand emphasis in dialogue remains unclear. This paper introduces Emphasized-Talk, a benchmark with emphasis-annotated dialogue samples capturing the implications of emphasis. We evaluate various LLMs, both open-source and commercial, to measure their performance in understanding emphasis. Additionally, we propose an automatic evaluation pipeline using GPT-4, which achieves a high correlation with human rating. Our findings reveal that although commercial LLMs generally perform better, there is still significant room for improvement in comprehending emphasized sentences.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
In Dialogue with Intelligence: Rethinking Large Language Models as Collective Knowledge
Large Language Models (LLMs) are typically analysed through architectural, behavioural, or training-data lenses. This article offers a theoretical and experiential re-framing: LLMs as dynamic instantiations of Collective…
From Personas to Talks: Revisiting the Impact of Personas on LLM-Synthesized Emotional Support Conversations
The rapid advancement of Large Language Models (LLMs) has revolutionized the generation of emotional support conversations (ESC), offering scalable solutions with reduced costs and enhanced data privacy. This paper explo…
DOGE: Towards Versatile Visual Document Grounding and Referring
In recent years, Multimodal Large Language Models (MLLMs) have increasingly emphasized grounding and referring capabilities to achieve detailed understanding and flexible user interaction. However, in the realm of visual…
document understandingIncorporate Dependency Relation Knowledge into Transformer Block for Multi-turn Dialogue Generation
Because of the compositionality of natural language, syntactic structure is one of the key factors for semantic understanding. However, the Transformer block, which is widely used for obtaining the distributed representa…
Dialogue GenerationRelationCutting Through the Confusion and Hype: Understanding the True Potential of Generative AI
This paper explores the nuanced landscape of generative AI (genAI), particularly focusing on neural network-based models like Large Language Models (LLMs). While genAI garners both optimistic enthusiasm and sceptical cri…