When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning

2026-06-02 · Ayushi Chadha arxiv

Long-horizon reasoning requires a system to commit to medium-horizon intent without becoming rigid: re-plan too often and computation never coheres into multi-step structure; commit too long and the plan goes stale. We study this stability-adaptivity tradeoff in the latent reasoning setting, where multi-step computation occurs inside hidden state rather than externalized token traces. We extend the Hierarchical Reasoning Model (HRM) with a feudal-style manager-worker interface: a slow high-level module periodically emits a normalized directional subgoal that persists for P low-level steps, biasing the worker's hidden-state updates and supplying an intrinsic cosine alignment loss. On ARC and ConceptARC, we find that subgoal persistence -- not subgoal injection alone -- is the central knob: moderate periods P in [3, 6] consistently outperform both very frequent (P=1) and very long horizons, with a clear minimum LM loss at P=3 (1.544 vs. 1.674 at P=1, 1.640 baseline; replicated over 5 seeds at mean 1.595, std 0.045). The intrinsic alignment weight lambda shows a complementary narrow optimum (lambda approximately 0.05). A controlled ablation at past-sweet-spot lambda isolates learned directional structure -- not architectural capacity or auxiliary loss alone -- as the source of interference when the alignment signal exceeds its optimum. Together these findings implicate a design principle for compositional planning in latent reasoning systems: medium-horizon intent must be coherent across enough computational steps for compositional structure to form.

📄 PDF Abstract BibTeX arXiv:2606.03741

Code (0)

등록된 구현이 없습니다.

Mind the Gap: Promises and Pitfalls of Hierarchical Planning in LeWorldModel

2026-07-14 · Niccolò Caselli, Salvatore Lo Sardo, Francesco Massafra, Ippokratis Pantelidis 외 arxiv

We investigate whether temporal hierarchy can improve LeWorldModel on long-horizon goal-conditioned control. We introduce Hi-LeWM, an extension that freezes the pretrained low-level LeWM and adds high-level planning over…

Probabilistic Subgoal Representations for Hierarchical Reinforcement learning

2024-06-24 · Vivienne Huiling Wang, Tinghuai Wang, Wenyan Yang, Joni-Kristian Kämäräinen 외

In goal-conditioned hierarchical reinforcement learning (HRL), a high-level policy specifies a subgoal for the low-level policy to reach. Effective HRL hinges on a suitable subgoal represen tation function, abstracting s…

Gaussian Processes Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning

Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

2023-07-22 · Qingyang Zhang, Yiming Yang, Jingqing Ruan, Xuantang Xiong 외

Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) is a promising paradigm to address the exploration-exploitation dilemma in reinforcement learning. It decomposes the source task into subgoal conditional subta…

continuous-control Continuous Control Hierarchical Reinforcement Learning reinforcement-learning+2

Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning

2025-06-26 · Jaebak Hwang, Sanghyeon Lee, Jeongmo Kim, Seungyul Han

Long-horizon goal-conditioned tasks pose fundamental challenges for reinforcement learning (RL), particularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial soluti…

Decision Making Hierarchical Reinforcement Learning Reinforcement Learning (RL)

Hierarchical Planning with Latent World Models

2026-04-03 · Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis 외 arxiv

World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponen…