Student-Initiated Action Advising via Advice Novelty

2020-10-01 · Ercument Ilhan, Jeremy Gow, Diego Perez-Liebana

Action advising is a budget-constrained knowledge exchange mechanism between teacher-student peers that can help tackle exploration and sample inefficiency problems in deep reinforcement learning (RL). Most recently, student-initiated techniques that utilise state novelty and uncertainty estimations have obtained promising results. However, the approaches built on these estimations have some potential weaknesses. First, they assume that the convergence of the student's RL model implies less need for advice. This can be misleading in scenarios with teacher absence early on where the student is likely to learn suboptimally by itself; yet also ignore the teacher's assistance later. Secondly, the delays between encountering states and having them to take effect in the RL model updates in presence of the experience replay dynamics cause a feedback lag in what the student actually needs advice for. We propose a student-initiated algorithm that alleviates these by employing Random Network Distillation (RND) to measure the novelty of a piece of advice. Furthermore, we perform RND updates only for the advised states to ensure that the student's own learning does not impair its ability to leverage the teacher. Experiments in GridWorld and MinAtar show that our approach performs on par with the state-of-the-art and demonstrates significant advantages in the scenarios where the existing methods are prone to fail.

📄 PDF Abstract BibTeX arXiv:2010.00381

Code (1)

ercumentilhan/advice-novelty 공식 구현 tf

Tasks

Atari GamesDeep Reinforcement LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Action Advising with Advice Imitation in Deep Reinforcement Learning

2021-04-17 · Ercument Ilhan, Jeremy Gow, Diego Perez-Liebana

Action advising is a peer-to-peer knowledge exchange technique built on the teacher-student paradigm to alleviate the sample inefficiency problem in deep reinforcement learning. Recently proposed student-initiated approa…

Atari Games Behavioural cloning Deep Reinforcement Learning reinforcement-learning+2

Explainable Action Advising for Multi-Agent Reinforcement Learning

2022-11-15 · Yue Guo, Joseph Campbell, Simon Stepputtis, Ruiyu Li 외

Action advising is a knowledge transfer technique for reinforcement learning based on the teacher-student paradigm. An expert teacher provides advice to a student during training in order to improve the student's sample …

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Methodical Advice Collection and Reuse in Deep Reinforcement Learning

2022-04-14 · Sahir, Ercüment İlhan, Srijita Das, Matthew E. Taylor

Reinforcement learning (RL) has shown great success in solving many challenging tasks via use of deep neural networks. Although using deep learning for RL brings immense representational power, it also causes a well-know…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Learning to Teach Reinforcement Learning Agents

2017-07-28 · Anestis Fachantidis, Matthew E. Taylor, Ioannis Vlahavas

In this article we study the transfer learning model of action advice under a budget. We focus on reinforcement learning teachers providing action advice to heterogeneous students playing the game of Pac-Man under a limi…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning

SmartCourse: A Contextual AI-Powered Course Advising System for Undergraduates

2025-07-26 · Yixuan Mi, Yiduo Yu, Yiyi Zhao arxiv

We present SmartCourse, an integrated course management and AI-driven advising system for undergraduate students (specifically tailored to the Computer Science (CPS) major). SmartCourse addresses the limitations of tradi…