Risk-Aware Distributional Intervention Policies for Language Models
Language models are prone to occasionally undesirable generations, such as harmful or toxic content, despite their impressive capability to produce texts that appear accurate and coherent. This paper presents a new two-stage approach to detect and mitigate undesirable content generations by rectifying activations. First, we train an ensemble of layerwise classifiers to detect undesirable content using activations by minimizing a smooth surrogate of the risk-aware score. Then, for contents that are detected as undesirable, we propose layerwise distributional intervention policies that perturb the attention heads minimally while guaranteeing probabilistically the effectiveness of the intervention. Benchmarks on several language models and datasets show that our method outperforms baselines in reducing the generation of undesirable output.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improving Robustness via Risk Averse Distributional Reinforcement Learning
One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Risk-Aware Reinforcement Learning for Mobile Manipulation
For robots to successfully transition from lab settings to everyday environments, they must begin to reason about the risks associated with their actions and make informed, risk-aware decisions. This is particularly true…
Reinforcement LearningA Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving
Reinforcement learning (RL) has demonstrated considerable potential for autonomous driving decision-making. However, its deployment in urban autonomous driving, particularly at highly interactive unsignalized intersectio…
Reinforcement LearningAutonomous DrivingUncertainty-aware Distributional Offline Reinforcement Learning
Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties asso…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Distributional Reinforcement Learning for Risk-Sensitive Policies
We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that applying the distributional Bellman optimality operator w…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)