Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns

2025-03-05 · Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, Gerhard Neumann

Soft Actor-Critic (SAC) critically depends on its critic network, which typically evaluates a single state-action pair to guide policy updates. Using N-step returns is a common practice to reduce the bias in the target values of the critic. However, using N-step returns can again introduce high variance and necessitates importance sampling, often destabilizing training. Recent algorithms have also explored action chunking-such as direct action repetition and movement primitives-to enhance exploration. In this paper, we propose a Transformer-based Critic Network for SAC that integrates the N-returns framework in a stable and efficient manner. Unlike approaches that perform chunking in the actor network, we feed chunked actions into the critic network to explore potential performance gains. Our architecture leverages the Transformer's ability to process sequential information, facilitating more robust value estimation. Empirical results show that this method not only achieves efficient, stable training but also excels in sparse reward/multi-phase environments-traditionally a challenge for step-based methods. These findings underscore the promise of combining Transformer-based critics with N-returns to advance reinforcement learning performance

📄 PDF Abstract BibTeX arXiv:2503.03660

Code (0)

등록된 구현이 없습니다.

Tasks

Chunking

Methods 이 논문이 사용한 방법론

Dilated Convolution 설명 없음

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Average Pooling 설명 없음

SAC 설명 없음

N-step Returns $n$-step Returns are used for value function estimation in reinforcement learning. Specifically, for $n$ steps we can write the complete return as: $$ R\_{t}^{(n)} =…

PAC-ACT: Post-training Actor-Critic for Action Chunking Transformers

2026-07-10 · Yujie Pang, Zudong Li arxiv

Precision industrial contact manipulation requires reliable robot policies under pose perturbations and contact-force constraints. Vision-language-action models offer broad generalization but often introduce high inferen…

ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

2026-05-10 · Qian Chen, Junqiao Zhao, Hongtu Zhou, Hang Yu 외 arxiv

Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic metho…

Reinforcement Learning Offline RL

Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

2026-05-07 · Nandiraju Gireesh, Yuanliang Ju, He Wang arxiv

Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is…

Reinforcement Learning

SEAR: Sample Efficient Action Chunking Reinforcement Learning

2026-03-02 · C. F. Maximilian Nagy, Onur Celik, Emiliyan Gospodinov, Florian Seligmann 외 arxiv

Action chunking improves exploration and accelerates value propagation in long-horizon reinforcement learning, but naively applying off-policy methods to the temporally extended action space at reduced decision frequency…