REALab: An Embedded Perspective on Tampering
This paper describes REALab, a platform for embedded agency research in reinforcement learning (RL). REALab is designed to model the structure of tampering problems that may arise in real-world deployments of RL. Standard Markov Decision Process (MDP) formulations of RL and simulated environments mirroring the MDP structure assume secure access to feedback (e.g., rewards). This may be unrealistic in settings where agents are embedded and can corrupt the processes producing feedback (e.g., human supervisors, or an implemented reward function). We describe an alternative Corrupt Feedback MDP formulation and the REALab environment platform, which both avoid the secure feedback assumption. We hope the design of REALab provides a useful perspective on tampering problems, and that the platform may serve as a unit test for the presence of tampering incentives in RL agent designs.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Similar Papers 제목 키워드 기반
A Novel Region Duplication Detection Algorithm Based on Hybrid Approach
The digital images from various sources are ubiquitous due to easy availability of high bandwidth Internet. Digital images are easy to tamper with good or bad intentions. Non-availability of pre-embedded information in d…
TamperNN: Efficient Tampering Detection of Deployed Neural Nets
Neural networks are powering the deployment of embedded devices and Internet of Things. Applications range from personal assistants to critical ones such as self-driving cars. It has been shown recently that models obtai…
QuantizationSelf-Driving CarsReward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective
Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)TAMPAR: Visual Tampering Detection for Parcel Logistics in Postal Supply Chains
Due to the steadily rising amount of valuable goods in supply chains, tampering detection for parcels is becoming increasingly important. In this work, we focus on the use-case last-mile delivery, where only a single RGB…
Change DetectionKeypoint DetectionEfficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)
Text embedding models play a crucial role in natural language processing, particularly in information retrieval, and their importance is further highlighted with the recent utilization of RAG (Retrieval- Augmented Genera…
Contrastive LearningInformation RetrievalRAGRetrieval+1