Diffusion-Based Offline RL for Improved Decision-Making in Augmented ARC Task

2024-10-15 · Yunho Kim, JaeHyun Park, Heejun Kim, Sejin Kim, Byung-Jun Lee, Sundong Kim

Effective long-term strategies enable AI systems to navigate complex environments by making sequential decisions over extended horizons. Similarly, reinforcement learning (RL) agents optimize decisions across sequences to maximize rewards, even without immediate feedback. To verify that Latent Diffusion-Constrained Q-learning (LDCQ), a prominent diffusion-based offline RL method, demonstrates strong reasoning abilities in multi-step decision-making, we aimed to evaluate its performance on the Abstraction and Reasoning Corpus (ARC). However, applying offline RL methodologies to enhance strategic reasoning in AI for solving tasks in ARC is challenging due to the lack of sufficient experience data in the ARC training set. To address this limitation, we introduce an augmented offline RL dataset for ARC, called Synthesized Offline Learning Data for Abstraction and Reasoning (SOLAR), along with the SOLAR-Generator, which generates diverse trajectory data based on predefined rules. SOLAR enables the application of offline RL methods by offering sufficient experience data. We synthesized SOLAR for a simple task and used it to train an agent with the LDCQ method. Our experiments demonstrate the effectiveness of the offline RL approach on a simple ARC task, showing the agent's ability to make multi-step sequential decisions and correctly identify answer states. These results highlight the potential of the offline RL approach to enhance AI's strategic reasoning capabilities.

📄 PDF Abstract BibTeX arXiv:2410.11324

Code (0)

등록된 구현이 없습니다.

Tasks

ARCDecision MakingNavigateOffline RLQ-LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation

2024-10-15 · JaeHyun Park, Yunho Kim, Sejin Kim, Byung-Jun Lee 외

We propose a novel offline reinforcement learning (offline RL) approach, introducing the Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation (DIAR) framework. We address two key challenges in offline RL:…

Decision Making Offline RL Q-Learning

Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning

2025-06-26 · Prajwal Koirala, Cody Fleming

Generative models such as diffusion and flow-matching offer expressive policies for offline reinforcement learning (RL) by capturing rich, multimodal action distributions, but their iterative sampling introduces high inf…

Action Generation Decision Making Offline RL Reinforcement Learning (RL)+1

Habitizing Diffusion Planning for Efficient and Effective Decision Making

2025-02-10 · Haofei Lu, Yifei Shen, Dongsheng Li, Junliang Xing 외

Diffusion models have shown great promise in decision-making, also known as diffusion planning. However, the slow inference speeds limit their potential for broader real-world applications. Here, we introduce Habi, a gen…

CPU D4RL Decision Making

Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning

2025-09-26 · Xianghua Zeng, Hao Peng, Angsheng Li, Yicheng Pan arxiv

Diffusion-based generative methods have shown promising potential for modeling trajectories from offline reinforcement learning (RL) datasets, and hierarchical diffusion has been introduced to mitigate variance accumulat…

Reinforcement Learning Trajectory Modeling Decision Making Offline RL

DiWA: Diffusion Policy Adaptation with World Models

2025-08-05 · Akshay L Chandra, Iman Nematollahi, Chenguang Huang, Tim Welschehold 외 arxiv

Fine-tuning diffusion policies with reinforcement learning (RL) presents significant challenges. The long denoising sequence for each action prediction impedes effective reward propagation. Moreover, standard RL methods …

Reinforcement Learning