InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

2024-09-12 · Andrew Lee, Ian Chuang, Ling-Yuan Chen, Iman Soltani

Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance. We provide supplementary materials and videos on our project page.

📄 PDF Abstract BibTeX arXiv:2409.07914

Code (0)

등록된 구현이 없습니다.

Tasks

Chunking DecoderImitation Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

The influence of Chunking on Dependency Crossing and Distance

2015-09-03 · Qian Lu, Chunshan Xu, Haitao Liu

This paper hypothesizes that chunking plays important role in reducing dependency distance and dependency crossings. Computer simulations, when compared with natural languages,show that chunking reduces mean dependency d…

Chunking

Investigating NP-Chunking with Universal Dependencies for English

2018-11-01 · WS 2018 11 · Oph{\'e}lie Lacroix

Chunking is a pre-processing task generally dedicated to improving constituency parsing. In this paper, we want to show that universal dependency (UD) parsing can also leverage the information provided by the task of chu…

Chunking Constituency Parsing Dependency Parsing Multi-Task Learning+3

Beyond Implicit Force: Evaluating Explicit Force-Torque Proxies in Action Chunking with Transformers

2026-07-16 · King Hang Wong, Lingqiao Liu, Feras Dayoub arxiv

Contact-rich manipulation requires policies to infer interaction state from signals that are often weakly observable through vision and kinematics alone. Action Chunking with Transformers (ACT) has shown strong performan…

Web Retrieval-Aware Chunking (W-RAC) for Efficient and Cost-Effective Retrieval-Augmented Generation Systems

2026-01-08 · Uday Allu, Sonu Kedia, Tanmay Odapally, Biddwan Ahmed arxiv

Retrieval-Augmented Generation (RAG) systems critically depend on effective document chunking strategies to balance retrieval quality, latency, and operational cost. Traditional chunking approaches, such as fixed-size, r…

Text Generation

SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation

2025-08-11 · Zeyu Yang, Lai Wei, Roman Koshkin, Xi Chen 외 arxiv

This work proposes a grammar-based chunking strategy that segments input streams into semantically complete units by parsing dependency relations (e.g., noun phrase boundaries, verb-object structures) and punctuation fea…