홈 › Papers

CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay

2024-02-07 · Natasha Butt, Blazej Manczak, Auke Wiggers, Corrado Rainone, David W. Zhang, Michaël Defferrard, Taco Cohen

Large language models are increasingly solving tasks that are commonly believed to require human-level reasoning ability. However, these models still perform very poorly on benchmarks of general intelligence such as the Abstraction and Reasoning Corpus (ARC). In this paper, we approach ARC as a programming-by-examples problem, and introduce a novel and scalable method for language model self-improvement called Code Iteration (CodeIt). Our method iterates between 1) program sampling and hindsight relabeling, and 2) learning from prioritized experience replay. By relabeling the goal of an episode (i.e., the target program output given input) to the realized output produced by the sampled program, our method effectively deals with the extreme sparsity of rewards in program synthesis. Applying CodeIt to the ARC dataset, we demonstrate that prioritized hindsight replay, along with pre-training and data-augmentation, leads to successful inter-task generalization. CodeIt is the first neuro-symbolic approach that scales to the full ARC evaluation dataset. Our method solves 15% of ARC evaluation tasks, achieving state-of-the-art performance and outperforming existing neural and symbolic baselines. Our code is available at https://github.com/Qualcomm-AI-research/codeit .

📄 PDF Abstract BibTeX arXiv:2402.04858

Code (1)

Qualcomm-AI-research/codeit 공식 구현 pytorch

Tasks

ARCData AugmentationLanguage ModelingLanguage Modelling Program Synthesis

Improvements on Hindsight Learning

2018-09-16 · Ameet Deshpande, Srikanth Sarma, Ashutosh Jha, Balaraman Ravindran

Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising w…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Advances in Experience Replay

2018-05-15 · Tracy Wan, Neil Xu

This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinat…

OpenAI Gym

Introspective Experience Replay: Look Back When Surprised

2022-06-07 · Ramnath Kumar, Dheeraj Nagaraj

In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience repla…

Q-Learning reinforcement-learning Reinforcement Learning (RL)

Regret Minimization Experience Replay in Off-Policy Reinforcement Learning

2021-05-15 · NeurIPS 2021 12 · Xu-Hui Liu, Zhenghai Xue, Jing-Cheng Pang, Shengyi Jiang 외

In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, rec…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Superior Performance with Diversified Strategic Control in FPS Games Using General Reinforcement Learning

2021-09-29 · Shuxing Li, Jiawei Xu, Chun Yuan, Peng Sun 외

This paper offers an overall solution for first-person shooter (FPS) games to achieve superior performance using general reinforcement learning (RL). We introduce an agent in ViZDoom that can surpass previous top agents …

FPS Games General Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)