Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Pretrained language models have demonstrated extraordinary capabilities in language generation. However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization. Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions. However, many important distributions, such as personal preferences, are unquantified. In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing Nano, a few-shot human-in-the-loop training algorithm that continuously learns from human feedback. Nano achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works. We also show that Nano is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals' personal preferences with high sample efficiency.
Code (1)
Tasks
AttributeFairnessLanguage ModelingLanguage ModellingText GenerationSimilar Papers 제목 키워드 기반
Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an…
counterfactualImitation LearningMuJoCoreinforcement-learning+2Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
We consider the task of estimating a structural model of dynamic decisions by a human agent based upon the observable history of implemented actions and visited states. This problem has an inherent nested structure: in t…
Imitation LearningMuJoCoStar Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
Training a family of large language models (LLMs), either from scratch or via iterative compression, is prohibitively expensive and inefficient, requiring separate training runs for each model in the family. In this pape…
Knowledge DistillationBiOpt: Bi-Level Optimization for Few-Shot Segmentation
Few-shot segmentation is a challenging task that aims to segment objects of new classes given scarce support images. In the inductive setting, existing prototype-based methods focus on extracting prototypes from the supp…
Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
Reinforcement learning often faces challenges with reward misalignment, where agents optimize for given rewards but fail to exhibit the desired behaviors. This occurs when the reward function incentivizes proxy behaviors…
continuous-controlContinuous Control