Agent Modeling as Auxiliary Task for Deep Reinforcement Learning

2019-07-22 · Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor

In this paper we explore how actor-critic methods in deep reinforcement learning, in particular Asynchronous Advantage Actor-Critic (A3C), can be extended with agent modeling. Inspired by recent works on representation learning and multiagent deep reinforcement learning, we propose two architectures to perform agent modeling: the first one based on parameter sharing, and the second one based on agent policy features. Both architectures aim to learn other agents' policies as auxiliary tasks, besides the standard actor (policy) and critic (values). We performed experiments in both cooperative and competitive domains. The former is a problem of coordinated multiagent object transportation and the latter is a two-player mini version of the Pommerman game. Our results show that the proposed architectures stabilize learning and outperform the standard A3C architecture when learning a best response in terms of expected rewards.

📄 PDF Abstract BibTeX arXiv:1907.09597

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

A3C A3C, Asynchronous Advantage Actor Critic, is a policy gradient algorithm in reinforcement learning that maintains a policy $\pi\left(a\_{t}\mid{s}\_{t}; \theta\right)$ and…

PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale

2022-10-15 · Kuang-Huei Lee, Ted Xiao, Adrian Li, Paul Wohlhart 외

The predictive information, the mutual information between the past and future, has been shown to be a useful representation learning auxiliary loss for training reinforcement learning agents, as the ability to model wha…

reinforcement-learning Reinforcement Learning (RL)Representation Learning Robot Manipulation

Policy and World Modeling Co-Training for Language Agents

2026-06-01 · Ning Lu, Baijiong Lin, Shengcai Liu, Jiahao Wu 외 arxiv

Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM)…

Reinforcement Learning

On The Effect of Auxiliary Tasks on Representation Dynamics

2021-02-25 · Clare Lyle, Mark Rowland, Georg Ostrovski, Will Dabney

While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understandin…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Discovery of Useful Questions as Auxiliary Tasks

2019-09-10 · NeurIPS 2019 12 · Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis 외

Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine lear…

Reinforcement Learning Reinforcement Learning (RL)

Generating Auxiliary Tasks with Reinforcement Learning

2025-10-27 · Judah Goldfeder, Matthew So, Hod Lipson arxiv

Auxiliary Learning (AL) is a form of multi-task learning in which a model trains on auxiliary tasks to boost performance on a primary objective. While AL has improved generalization across domains such as navigation, ima…

Reinforcement Learning Image Classification Multi-Task Learning