Programmatic Policy Extraction by Iterative Local Search
Reinforcement learning policies are often represented by neural networks, but programmatic policies are preferred in some cases because they are more interpretable, amenable to formal verification, or generalize better. While efficient algorithms for learning neural policies exist, learning programmatic policies is challenging. Combining imitation-projection and dataset aggregation with a local search heuristic, we present a simple and direct approach to extracting a programmatic policy from a pretrained neural policy. After examining our local search heuristic on a programming by example problem, we demonstrate our programmatic policy extraction method on a pendulum swing-up problem. Both when trained using a hand crafted expert policy and a learned neural policy, our method discovers simple and interpretable policies that perform almost as well as the original.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training
Multi-agent reinforcement learning (MARL) has achieved significant progress in solving complex multi-player games through self-play. However, training effective adversarial policies requires millions of experience sample…
Multi-agent Reinforcement LearningProgrammatically Interpretable Reinforcement Learning
We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinfo…
Car RacingDeep Reinforcement Learningreinforcement-learningReinforcement Learning+1Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks
Programmatically Interpretable Reinforcement Learning (PIRL) encodes policies in human-readable computer programs. Novel algorithms were recently introduced with the goal of handling the lack of gradient signal to guide …
TranslationReclaiming the Source of Programmatic Policies: Programmatic versus Latent Spaces
Recent works have introduced LEAPS and HPRL, systems that learn latent spaces of domain-specific languages, which are used to define programmatic policies for partially observable Markov decision processes (POMDPs). Thes…
Programmatic Reinforcement Learning without Oracles
Deep reinforcement learning (RL) has led to encouraging successes in many challenging control tasks. However, a deep RL model lacks interpretability due to the difficulty of identifying how the model's control logic rela…
Bilevel OptimizationDeep Reinforcement LearningPolicy Gradient Methodsreinforcement-learning+2