Return-based Scaling: Yet Another Normalisation Trick for Deep RL
Scaling issues are mundane yet irritating for practitioners of reinforcement learning. Error scales vary across domains, tasks, and stages of learning; sometimes by many orders of magnitude. This can be detrimental to learning speed and stability, create interference between learning tasks, and necessitate substantial tuning. We revisit this topic for agents based on temporal-difference learning, sketch out some desiderata and investigate scenarios where simple fixes fall short. The mechanism we propose requires neither tuning, clipping, nor adaptation. We validate its effectiveness and robustness on the suite of Atari games. Our scaling method turns out to be particularly helpful at mitigating interference, when training a shared neural network on multiple targets that differ in reward scale or discounting.
Code (0)
등록된 구현이 없습니다.
Tasks
Atari Gamesreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
The Affine Divergence: Aligning Activation Updates Beyond Normalisation
A systematic mismatch exists between mathematically ideal and effective activation updates during gradient descent. As intended, parameters update in their direction of steepest descent. However, activations are argued t…
Normalisation of Weights and Firing Rates in Spiking Neural Networks with Spike-Timing-Dependent Plasticity
Maintaining the ability to fire sparsely is crucial for information encoding in neural networks. Additionally, spiking homeostasis is vital for spiking neural networks with changing numbers of weights and neurons. We dis…
ClusteringTime SeriesTime Series AnalysisA Systematic Benchmark of Intensity Normalisation Methods for 3D Knee MRI Segmentation and Cross-Domain Generalisability
Robust out-of-the-box performance is essential for the clinical deployment of deep learning models in medical imaging. An important but underexplored factor affecting model generalisability is intensity normalisation, pa…
An Exact Finite-dimensional Explicit Feature Map for Kernel Functions
Kernel methods in machine learning use a kernel function that takes two data points as input and returns their inner product after mapping them to a Hilbert space, implicitly and without actually computing the mapping. F…
Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
Large language model (LLM) scaling is hitting a wall. Widening models yields diminishing returns, and extending context length does not improve fundamental expressivity. In contrast, depth scaling offers theoretically su…