Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning

2024-10-31 · Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, Sanjit A. Seshia

Goal-conditioned reinforcement learning is a powerful way to control an AI agent's behavior at runtime. That said, popular goal representations, e.g., target states or natural language, are either limited to Markovian tasks or rely on ambiguous task semantics. We propose representing temporal goals using compositions of deterministic finite automata (cDFAs) and use cDFAs to guide RL agents. cDFAs balance the need for formal temporal semantics with ease of interpretation: if one can understand a flow chart, one can understand a cDFA. On the other hand, cDFAs form a countably infinite concept class with Boolean semantics, and subtle changes to the automaton can result in very different tasks, making them difficult to condition agent behavior on. To address this, we observe that all paths through a DFA correspond to a series of reach-avoid tasks and propose pre-training graph neural network embeddings on "reach-avoid derived" DFAs. Through empirical evaluation, we demonstrate that the proposed pre-training method enables zero-shot generalization to various cDFA task classes and accelerated policy specialization without the myopic suboptimality of hierarchical methods.

📄 PDF Abstract BibTeX arXiv:2411.00205

Code (0)

등록된 구현이 없습니다.

Tasks

Graph Neural Network reinforcement-learning Reinforcement Learning Zero-shot Generalization

Methods 이 논문이 사용한 방법론

Graph Neural Network 설명 없음

DFA 설명 없음

Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learning

2025-03-06 · Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, Sanjit A. Seshia

Automata-conditioned reinforcement learning (RL) has given promising results for learning multi-task policies capable of performing temporally extended objectives given at runtime, done by pretraining and freezing automa…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning

2025-10-16 · Roger Creus Castanyer, Faisal Mohamed, Pablo Samuel Castro, Cyrus Neary 외 arxiv

Reinforcement learning (RL) algorithms are highly sensitive to reward function specification, which remains a central challenge limiting their broad applicability. We present ARM-FM: Automated Reward Machines via Foundat…

Zero-shot Generalization Reinforcement Learning

Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning

2025-06-23 · Anthony Kobanda, Waris Radji, Mathieu Petitbois, Odalric-Ambrym Maillard 외

Offline Goal-Conditioned Reinforcement Learning seeks to train agents to reach specified goals from previously collected trajectories. Scaling that promises to long-horizon tasks remains challenging, notably due to compo…

Metric Learning reinforcement-learning Reinforcement Learning

Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning

2026-05-20 · Junseok Kim, Dohyeong Kim, Mineui Hong, Songhwai Oh arxiv

Compositional generalization is essential for reaching unseen goals under novel contextual variations in offline goal-conditioned reinforcement learning (GCRL), where a generalist goal-reaching agent must be learned from…

Reinforcement Learning

Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning

2025-11-04 · Beyazit Yalcinkaya, Marcell Vazquez-Chanlatte, Ameesh Shah, Hanna Krasowski 외 arxiv

We study learning multi-task, multi-agent policies for cooperative, temporal objectives, under centralized training, decentralized execution. In this setting, using automata to represent tasks assigned to agents enables …

Multi-agent Reinforcement Learning