An Efficient Continuous Control Perspective for Reinforcement-Learning-based Sequential Recommendation
Sequential recommendation, where user preference is dynamically inferred from sequential historical behaviors, is a critical task in recommender systems (RSs). To further optimize long-term user engagement, offline reinforcement-learning-based RSs have become a mainstream technique as they provide an additional advantage in avoiding global explorations that may harm online users' experiences. However, previous studies mainly focus on discrete action and policy spaces, which might have difficulties in handling dramatically growing items efficiently. To mitigate this issue, in this paper, we aim to design an algorithmic framework applicable to continuous policies. To facilitate the control in the low-dimensional but dense user preference space, we propose an \underline{\textbf{E}}fficient \underline{\textbf{Co}}ntinuous \underline{\textbf{C}}ontrol framework (ECoC). Based on a statistically tested assumption, we first propose the novel unified action representation abstracted from normalized user and item spaces. Then, we develop the corresponding policy evaluation and policy improvement procedures. During this process, strategic exploration and directional control in terms of unified actions are carefully designed and crucial to final recommendation decisions. Moreover, beneficial from unified actions, the conservatism regularization for policies and value functions are combined and perfectly compatible with the continuous framework. The resulting dual regularization ensures the successful offline training of RL-based recommendation policies. Finally, we conduct extensive experiments to validate the effectiveness of our framework. The results show that compared to the discrete baselines, our ECoC is trained far more efficiently. Meanwhile, the final policies outperform baselines in both capturing the offline data and gaining long-term rewards.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlRecommendation SystemsSequential RecommendationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SS4Rec: Continuous-Time Sequential Recommendation with State Space Models
Sequential recommendation is a key area in the field of recommendation systems aiming to model user interest based on historical interaction sequences with irregular intervals. While previous recurrent neural network-bas…
Recommendation SystemsSequential RecommendationState Space ModelsA Survey on Sequential Recommendation
Different from most conventional recommendation problems, sequential recommendation focuses on learning users' preferences by exploiting the internal order and dependency among the interacted items, which has received si…
Sequential RecommendationSurveyA New Perspective on Transformers in Online Reinforcement Learning for Continuous Control
Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design dec…
Reinforcement LearningContinuous ControlLearning Graph ODE for Continuous-Time Sequential Recommendation
Sequential recommendation aims at understanding user preference by capturing successive behavior correlations, which are usually represented as the item purchasing sequences based on their past interactions. Existing eff…
Sequential RecommendationContinual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction
Deep reinforcement learning has made significant progress in the field of continuous control, such as physical control and autonomous driving. However, it is challenging for a reinforcement model to learn a policy for ea…
Autonomous Drivingcontinuous-controlContinuous ControlDeep Reinforcement Learning+4