Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

2018-07-06 · NeurIPS 2018 12 · Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, Ni Lao

We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate. MAPO is applicable to deterministic environments with discrete actions, such as structured prediction and combinatorial optimization tasks. We express the expected return objective as a weighted sum of two terms: an expectation over the high-reward trajectories inside the memory buffer, and a separate expectation over trajectories outside the buffer. To make an efficient algorithm of MAPO, we propose: (1) memory weight clipping to accelerate and stabilize training; (2) systematic exploration to discover high-reward trajectories; (3) distributed sampling from inside and outside of the memory buffer to scale up training. MAPO improves the sample efficiency and robustness of policy gradient, especially on tasks with sparse rewards. We evaluate MAPO on weakly supervised program synthesis from natural language (semantic parsing). On the WikiTableQuestions benchmark, we improve the state-of-the-art by 2.6%, achieving an accuracy of 46.3%. On the WikiSQL benchmark, MAPO achieves an accuracy of 74.9% with only weak supervision, outperforming several strong baselines with full supervision. Our source code is available at https://github.com/crazydonkey200/neural-symbolic-machines

📄 PDF Abstract BibTeX arXiv:1807.02322

Code (4)

crazydonkey200/neural-symbolic-machines 공식 구현 tf

niansong1996/wassp tf

pcyin/pytorch_neural_symbolic_machines pytorch

theSparta/neural-symbolic-machines tf

Tasks

Combinatorial OptimizationProgram Synthesis Semantic Parsing Structured Prediction

Towards Mixed Optimization for Reinforcement Learning with Program Synthesis

2018-07-01 · Surya Bhupatiraju, Kumar Krishna Agrawal, Rishabh Singh

Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification co…

Deep Reinforcement Learning Program Repair Program Synthesis reinforcement-learning+2

Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis

2025-10-01 · Carlo Bosio, Matteo Guarrera, Alberto Sangiovanni-Vincentelli, Mark W. Mueller arxiv

Large Language models (LLMs) have shown promise as generators of symbolic control policies, producing interpretable program-like representations through iterative search. However, these models are not capable of separati…

Program Synthesis

QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management

2025-12-15 · Weizhou Shen, Ziyi Yang, Chenliang Li, Zhiyuan Lu 외 arxiv

We introduce QwenLong-L1.5, a model that achieves superior long-context reasoning capabilities through systematic post-training innovations. The key technical breakthroughs of QwenLong-L1.5 are as follows: (1) Long-Conte…

Reinforcement Learning

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

2026-02-26 · Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li 외 arxiv

Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel sta…

Reinforcement Learning

Hexcute: A Tile-based Programming Language with Automatic Layout and Task-Mapping Synthesis

2025-04-22 · Xiao Zhang, Yaoyao Ding, Yang Hu, Gennady Pekhimenko

Deep learning (DL) workloads mainly run on accelerators like GPUs. Recent DL quantization techniques demand a new matrix multiplication operator with mixed input data types, further complicating GPU optimization. Prior h…

GPU Quantization