POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling
Conversational search systems, such as Google Assistant and Microsoft Cortana, provide a new search paradigm where users are allowed, via natural language dialogues, to communicate with search systems. Evaluating such systems is very challenging since search results are presented in the format of natural language sentences. Given the unlimited number of possible responses, collecting relevance assessments for all the possible responses is infeasible. In this paper, we propose POSSCORE, a simple yet effective automatic evaluation method for conversational search. The proposed embedding-based metric takes the influence of part of speech (POS) of the terms in the response into account. To the best knowledge, our work is the first to systematically demonstrate the importance of incorporating syntactic information, such as POS labels, for conversational search evaluation. Experimental results demonstrate that our metrics can correlate with human preference, achieving significant improvements over state-of-the-art baseline metrics.
Code (1)
Tasks
Conversational SearchPOSSimilar Papers 제목 키워드 기반
Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search
Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limi…
Conversational SearchText GenerationA Conceptual Framework for Implicit Evaluation of Conversational Search Interfaces
Conversational search (CS) has recently become a significant focus of the information retrieval (IR) research community. Multiple studies have been conducted which explore the concept of conversational search. Understand…
Conversational SearchInformation RetrievalRetrievalInterpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embedding
Conversational dense retrieval has shown to be effective in conversational search. However, a major limitation of conversational dense retrieval is their lack of interpretability, hindering intuitive understanding of mod…
Conversational SearchRetrievalConvSDG: Session Data Generation for Conversational Search
Conversational search provides a more convenient interface for users to search by allowing multi-turn interaction with the search engine. However, the effectiveness of the conversational dense retrieval methods is limite…
Conversational SearchRetrievalText GenerationChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval
Conversational search requires accurate interpretation of user intent from complex multi-turn contexts. This paper presents ChatRetriever, which inherits the strong generalization capability of large language models to r…
Contrastive LearningConversational SearchRetrieval