User Tampering in Reinforcement Learning Recommender Systems

2021-09-09 · Charles Evans, Atoosa Kasirzadeh

In this paper, we introduce new formal methods and provide empirical evidence to highlight a unique safety concern prevalent in reinforcement learning (RL)-based recommendation algorithms -- 'user tampering.' User tampering is a situation where an RL-based recommender system may manipulate a media user's opinions through its suggestions as part of a policy to maximize long-term user engagement. We use formal techniques from causal modeling to critically analyze prevailing solutions proposed in the literature for implementing scalable RL-based recommendation systems, and we observe that these methods do not adequately prevent user tampering. Moreover, we evaluate existing mitigation strategies for reward tampering issues, and show that these methods are insufficient in addressing the distinct phenomenon of user tampering within the context of recommendations. We further reinforce our findings with a simulation study of an RL-based recommendation system focused on the dissemination of political content. Our study shows that a Q-learning algorithm consistently learns to exploit its opportunities to polarize simulated users with its early recommendations in order to have more consistent success with subsequent recommendations that align with this induced polarization. Our findings emphasize the necessity for developing safer RL-based recommendation systems and suggest that achieving such safety would require a fundamental shift in the design away from the approaches we have seen in the recent literature.

📄 PDF Abstract BibTeX arXiv:2109.04083

Code (0)

등록된 구현이 없습니다.

Tasks

Q-LearningRecommendation Systems reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Explicit User Manipulation in Reinforcement Learning Based Recommender Systems

2022-03-20 · Matthew Sparr

Recommender systems are highly prevalent in the modern world due to their value to both users and platforms and services that employ them. Generally, they can improve the user experience and help to increase satisfaction…

Recommendation Systems reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems

2023-08-22 · Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach 외

Reinforcement learning serves as a potent tool for modeling dynamic user interests within recommender systems, garnering increasing research attention of late. However, a significant drawback persists: its poor data effi…

Recommendation Systems reinforcement-learning Reinforcement Learning

Handling Cold-Start Collaborative Filtering with Reinforcement Learning

2018-06-16 · Hima Varsha Dureddy, Zachary Kaden

A major challenge in recommender systems is handling new users, whom are also called $\textit{cold-start}$ users. In this paper, we propose a novel approach for learning an optimal series of questions with which to inter…

Collaborative Filtering Recommendation Systems reinforcement-learning Reinforcement Learning+1

UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution

2024-01-12 · Gengrui Zhang, Yao Wang, Xiaoshuang Chen, Hongyi Qian 외

In recent years, there has been a growing interest in utilizing reinforcement learning (RL) to optimize long-term rewards in recommender systems. Since industrial recommender systems are typically designed as multi-stage…

Multi-agent Reinforcement Learning Recommendation Systems reinforcement-learning Reinforcement Learning+1

Deep Reinforcement Learning for List-wise Recommendations

2017-12-30 · Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding 외

Recommender systems play a crucial role in mitigating the problem of information overload by suggesting users' personalized items or services. The vast majority of traditional recommender systems consider the recommendat…

Deep Reinforcement Learning Recommendation Systems reinforcement-learning Reinforcement Learning+1