Importance of Search and Evaluation Strategies in Neural Dialogue Modeling
We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher rated conversations. However, finding the optimal selection mechanism to choose from a more diverse set of candidates is still an open question.
Code (1)
Tasks
Dialogue GenerationDiversityOpen-Ended Question AnsweringSimilar Papers 제목 키워드 기반
RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
Reward modeling has become a cornerstone of aligning large language models (LLMs) with human preferences. Yet, when extended to subjective and open-ended domains such as role play, existing reward models exhibit severe d…
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
User Satisfaction Modeling (USM) is one of the popular choices for task-oriented dialogue systems evaluation, where user satisfaction typically depends on whether the user's task goals were fulfilled by the system. Task-…
AttributeLanguage ModellingTask-Oriented Dialogue SystemsPredicting Evoked Emotions in Conversations
Understanding and predicting the emotional trajectory in multi-party multi-turn conversations is of great significance. Such information can be used, for example, to generate empathetic response in human-machine interact…
Towards Mining Effective Pedagogical Strategies from Learner-LLM Educational Dialogues
Dialogue plays a crucial role in educational settings, yet existing evaluation methods for educational applications of large language models (LLMs) primarily focus on technical performance or learning outcomes, often neg…
Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation
Synthetic data is increasingly critical for contact centers, where privacy constraints and data scarcity limit the availability of real conversations. However, generating synthetic dialogues that are realistic and useful…
Dialogue Generation