Avoiding Tampering Incentives in Deep RL via Decoupled Approval
How can we design agents that pursue a given objective when all feedback mechanisms are influenceable by the agent? Standard RL algorithms assume a secure reward function, and can thus perform poorly in settings where agents can tamper with the reward-generating mechanism. We present a principled solution to the problem of learning from influenceable feedback, which combines approval with a decoupled feedback collection procedure. For a natural class of corruption functions, decoupled approval algorithms have aligned incentives both at convergence and for their local updates. Empirically, they also scale to complex 3D environments where tampering is possible.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
REALab: An Embedded Perspective on Tampering
This paper describes REALab, a platform for embedded agency research in reinforcement learning (RL). REALab is designed to model the structure of tampering problems that may arise in real-world deployments of RL. Standar…
Reinforcement Learning (RL)Approval Voting and Incentives in Crowdsourcing
The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not expert…
Incentives to Offer Algorithmic Recourse
Due to the importance of artificial intelligence (AI) in a variety of high-stakes decisions, such as loan approval, job hiring, and criminal bail, researchers in Explainable AI (XAI) have developed algorithms to provide …
Explainable Artificial Intelligence (XAI)MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking
Future advanced AI systems may learn sophisticated strategies through reinforcement learning (RL) that humans cannot understand well enough to safely evaluate. We propose a training method which avoids agents learning un…
Reinforcement Learning (RL)Causative Cyberattacks on Online Learning-based Automated Demand Response Systems
Power utilities are adopting Automated Demand Response (ADR) to replace the costly fuel-fired generators and to preempt congestion during peak electricity demand. Similarly, third-party Demand Response (DR) aggregators a…