Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation
In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models.
Code (1)
Tasks
Text GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tun…
GPUReinforcement Learning (RL)Dual Action Policy for Robust Sim-to-Real Reinforcement Learning
This paper presents Dual Action Policy (DAP), a novel approach to address the dynamics mismatch inherent in the sim-to-real gap of reinforcement learning. DAP uses a single policy to predict two sets of actions: one for …
Domain AdaptationReward-Conditioned Reinforcement Learning
Single-task RL agents are typically trained under a fixed reward function, which limits their robustness to reward misspecification and their ability to adapt to changing preferences. We introduce Reward-Conditioned Rein…
Reinforcement LearningMulti-Task LearningReward-Relevance-Filtered Linear Offline Reinforcement Learning
This paper studies offline reinforcement learning with linear function approximation in a setting with decision-theoretic, but not estimation sparsity. The structural restrictions of the data-generating process presume t…
reinforcement-learningReinforcement LearningHindsight Reward Tweaking via Conditional Deep Reinforcement Learning
Designing optimal reward functions has been desired but extremely difficult in reinforcement learning (RL). When it comes to modern complex tasks, sophisticated reward functions are widely used to simplify policy learnin…
Deep Reinforcement LearningMuJoCoreinforcement-learningReinforcement Learning+1