Emergence of In-Context Reinforcement Learning from Noise Distillation
Recently, extensive studies in Reinforcement Learning have been carried out on the ability of transformers to adapt in-context to various environments and tasks. Current in-context RL methods are limited by their strict requirements for data, which needs to be generated by RL agents or labeled with actions from an optimal policy. In order to address this prevalent problem, we propose AD$^\varepsilon$, a new data acquisition approach that enables in-context Reinforcement Learning from noise-induced curriculum. We show that it is viable to construct a synthetic noise injection curriculum which helps to obtain learning histories. Moreover, we experimentally demonstrate that it is possible to alleviate the need for generation using optimal policies, with in-context RL still able to outperform the best suboptimal policy in a learning dataset by a 2x margin.
Code (1)
Tasks
In-Context Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Representation Convergence: Mutual Distillation is Secretly a Form of Regularization
In this paper, we argue that mutual distillation between reinforcement learning policies serves as an implicit regularization, preventing them from overfitting to irrelevant features. We highlight two key contributions: …
Deep Reinforcement LearningFormreinforcement-learningReinforcement Learning+3GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
Reinforcement learning (RL) can be used to improve the policy (denoiser) of diffusion large language models (dLLMs), while being hindered by the intractability of the policy likelihood. A dominant and efficient family of…
Reinforcement LearningIn-context Reinforcement Learning with Algorithm Distillation
We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats…
In-Context Reinforcement Learningreinforcement-learningReinforcement LearningTED: Training-Free Experience Distillation for Multimodal Reasoning
Knowledge distillation is typically realized by transferring a teacher model's knowledge into a student's parameters through supervised or reinforcement-based optimization. While effective, such approaches require repeat…
Knowledge DistillationMultimodal ReasoningConvergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context. Recent empirical studies further demons…
Reinforcement Learning