The Phenomenon of Policy Churn

2022-06-01 · Tom Schaul, André Barreto, John Quan, Georg Ostrovski

We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs to just a handful, all related to deep learning. Finally, we hypothesise that policy churn is a beneficial but overlooked form of implicit exploration that casts $\epsilon$-greedy exploration in a fresh light, namely that $\epsilon$-noise plays a much smaller role than expected.

📄 PDF Abstract BibTeX arXiv:2206.00730

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn

2024-09-07 · Hongyao Tang, Glen Berseth

Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature…

Deep Reinforcement Learning Reinforcement Learning (RL)

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

2026-08-24 · Jingjie Ning, Xueqi Li arxiv

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are hel…

Natural Questions

Reducing Model Churn: Stable Re-training of Conversational Agents

2022-09-01 · SIGDIAL (ACL) 2022 9 · Christopher Hidey, Fei Liu, Rahul Goel

Retraining modern deep learning systems can lead to variations in model performance even when trained using the same data and hyper-parameters by simply using different random seeds. This phenomenon is known as model chu…

Semantic Parsing Structured Prediction

Augmenting Intelligence: A Hybrid Framework for Scalable and Stable Explanations

2025-12-22 · Lawrence Krukrubo, Julius Odede, Olawande Olusegun arxiv

Current approaches to Explainable AI (XAI) face a "Scalability-Stability Dilemma." Post-hoc methods (e.g., LIME, SHAP) may scale easily but suffer from instability, while supervised explanation frameworks (e.g., TED) off…

On Analyzing Churn Prediction in Mobile Games

2021-04-12 · Kihoon Jang, Junwhan Kim, Byunggu Yu

In subscription-based businesses, the churn rate refers to the percentage of customers who discontinue their subscriptions within a given time period. Particularly, in the mobile games industry, the churn rate is often p…

Prediction