System-Level Natural Language Feedback
Natural language (NL) feedback offers rich insights into user experience. While existing studies focus on an instance-level approach, where feedback is used to refine specific examples, we introduce a framework for system-level use of NL feedback. We show how to use feedback to formalize system-level design decisions in a human-in-the-loop-process -- in order to produce better models. In particular this is done through: (i) metric design for tasks; and (ii) language model prompt design for refining model responses. We conduct two case studies of this approach for improving search query and dialog response generation, demonstrating the effectiveness of system-level feedback. We show the combination of system-level and instance-level feedback brings further gains, and that human written instance-level feedback results in more grounded refinements than GPT-3.5 written ones, underlying the importance of human feedback for building systems. We release our code and data at https://github.com/yyy-Apple/Sys-NL-Feedback.
Code (1)
Tasks
Language ModelingLanguage ModellingResponse GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback
We study the task of semantic parse correction with natural language feedback. Given a natural language utterance, most semantic parsing systems pose the problem as one-shot translation where the utterance is mapped to a…
FormSemantic ParsingText to SQLText-To-SQL+1RL with Learnable Textual Feedback: A Bilevel Approach
Reinforcement learning with verifiable rewards can improve LLM reasoning, but learning remains sample-inefficient when terminal rewards are sparse. This has motivated a growing line of work on RL with textual feedback, w…
Reinforcement LearningEditScribe: Non-Visual Image Editing with Natural Language Verification Loops
Image editing is an iterative process that requires precise visual evaluation and manipulation for the output to match the editing intent. However, current image editing tools do not provide accessible interaction nor su…
Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leavi…
Reinforcement LearningA Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback
Interactive search can provide a better experience by incorporating interaction feedback from the users. This can significantly improve search accuracy as it helps avoid irrelevant information and captures the users' sea…
Deep Reinforcement LearningQ-LearningReinforcement Learning (RL)Sentence