CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.
Code (1)
Tasks
Contrastive Learningreinforcement-learningReinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions
Large language models (LLMs) must often respond to highly ambiguous user requests. In such cases, the LLM's best response may be to ask a clarifying question to elicit more information. We observe existing LLMs often res…
Untangling Braids with Multi-agent Q-Learning
We use reinforcement learning to tackle the problem of untangling braids. We experiment with braids with 2 and 3 strands. Two competing players learn to tangle and untangle a braid. We interface the braid untangling prob…
OpenAI GymQ-Learningreinforcement-learningReinforcement Learning+1Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification
Real-world visual question answering (VQA) is often context-dependent: an image-question pair may be under-specified, such that the correct answer depends on external information that is not observable in the image. In s…
Visual Question AnsweringReinforcement LearningQuestion GenerationPseudo Ambiguous and Clarifying Questions Based on Sentence Structures Toward Clarifying Question Answering System
Question answering (QA) with disambiguation questions is essential for practical QA systems because user questions often do not contain information enough to find their answers. We call this task clarifying question answ…
Question AnsweringQuestion GenerationQuestion-GenerationSentenceSecrets of RLHF in Large Language Models Part II: Reward Modeling
Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward mode…
Contrastive LearningMeta-Learningreinforcement-learning