Perspectives for Evaluating Conversational AI
Conversational AI systems are becoming famous in day to day lives. In this paper, we are trying to address the following key question: To identify whether design, as well as development efforts for search oriented conversational AI are successful or not.It is tricky to define 'success' in the case of conversational AI and equally tricky part is to use appropriate metrics for the evaluation of conversational AI. We propose four different perspectives namely user experience, information retrieval, linguistic and artificial intelligence for the evaluation of conversational AI systems. Additionally, background details of conversational AI systems are provided including desirable characteristics of personal assistants, differences between chatbot and an AI based personal assistant. An importance of personalization and how it can be achieved is explained in detail. Current challenges in the development of an ideal conversational AI (personal assistant) are also highlighted along with guidelines for achieving personalized experience for users.
Code (0)
등록된 구현이 없습니다.
Tasks
ChatbotInformation RetrievalRetrievalSimilar Papers 제목 키워드 기반
Pitfalls of Conversational LLMs on News Debiasing
This paper addresses debiasing in news editing and evaluates the effectiveness of conversational Large Language Models in this task. We designed an evaluation checklist tailored to news editors' perspectives, obtained ge…
MisinformationMeta-evaluation of Conversational Search Evaluation Metrics
Conversational search systems, such as Google Assistant and Microsoft Cortana, enable users to interact with search systems in multiple rounds through natural language dialogues. Evaluating such systems is very challengi…
Conversational SearchInformativenessAn Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives
Mental health conversational agents (a.k.a. chatbots) are widely studied for their potential to offer accessible support to those experiencing mental health challenges. Previous surveys on the topic primarily consider pa…
EthicsExperimental DesignMemory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents
Prior research on memory mechanism in RAG-based conversational system has emphasized how memory is stored and retrieved. However, far less is known about how memories with different functional roles influence response qu…
Old Habits Die Hard: How Conversational History Geometrically Traps LLMs
How does the conversational past of large language models (LLMs) influence their future performance? Recent work suggests that LLMs are affected by their conversational history in unexpected ways. For instance, hallucina…