Reinforcement Learning with Automated Auxiliary Loss Search

2022-10-12 · Tairan He, Yuge Zhang, Kan Ren, Minghuan Liu, Che Wang, Weinan Zhang, Yuqing Yang, Dongsheng Li

A good state representation is crucial to solving complicated reinforcement learning (RL) challenges. Many recent works focus on designing auxiliary losses for learning informative representations. Unfortunately, these handcrafted objectives rely heavily on expert knowledge and may be sub-optimal. In this paper, we propose a principled and universal method for learning better representations with auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS), which automatically searches for top-performing auxiliary loss functions for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size $7.5 \times 10^{20}$ and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss (namely, A2-winner) significantly improves the performance on both high-dimensional (image) and low-dimensional (vector) unseen tasks with much higher efficiency, showing promising generalization ability to different settings and even different benchmark domains. We conduct a statistical analysis to reveal the relations between patterns of auxiliary losses and RL performance.

📄 PDF Abstract BibTeX arXiv:2210.06041

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

AARL: Automated Auxiliary Loss for Reinforcement Learning

2021-09-29 · Tairan He, Yuge Zhang, Kan Ren, Che Wang 외

A good state representation is crucial to reinforcement learning (RL) while an ideal representation is hard to learn only with signals from the RL objective. Thus, many recent works manually design auxiliary losses to im…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectives

2024-11-29 · Armin Saghafian, Amirmohammad Izadi, Negin Hashemi Dijujin, Mahdieh Soleymani Baghshah

Grounding the instruction in the environment is a key step in solving language-guided goal-reaching reinforcement learning problems. In automated reinforcement learning, a key concern is to enhance the model's ability to…

reinforcement-learning Reinforcement Learning Retrieval Systematic Generalization+2

Adapting Auxiliary Losses Using Gradient Similarity

2018-12-05 · Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Mehrdad Farajtabar 외

One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be h…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Intrinsically Motivated Self-supervised Learning in Reinforcement Learning

2021-06-26 · Yue Zhao, Chenzhuang Du, Hang Zhao, Tiejun Li

In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency. However, abu…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

Loss is its own Reward: Self-Supervision for Reinforcement Learning

2016-12-21 · Evan Shelhamer, Parsa Mahmoudieh, Max Argus, Trevor Darrell

Reinforcement learning optimizes policies for expected cumulative reward. Need the supervision be so narrow? Reward is delayed and sparse for many tasks, making it a difficult and impoverished signal for end-to-end optim…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning