홈 › Papers

Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation

2024-09-26 · Quanting Xie, So Yeon Min, Pengliang Ji, Yue Yang, Tianyi Zhang, Kedi Xu, Aarav Bajaj, Ruslan Salakhutdinov, Matthew Johnson-Roberson, Yonatan Bisk

There is no limit to how much a robot might explore and learn, but all of that knowledge needs to be searchable and actionable. Within language research, retrieval augmented generation (RAG) has become the workhorse of large-scale non-parametric knowledge; however, existing techniques do not directly transfer to the embodied domain, which is multimodal, where data is highly correlated, and perception requires abstraction. To address these challenges, we introduce Embodied-RAG, a framework that enhances the foundational model of an embodied agent with a non-parametric memory system capable of autonomously constructing hierarchical knowledge for both navigation and language generation. Embodied-RAG handles a full range of spatial and semantic resolutions across diverse environments and query types, whether for a specific object or a holistic description of ambiance. At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail. This hierarchical organization allows the system to efficiently generate context-sensitive outputs across different robotic platforms. We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries across kilometer-level environments, highlighting its promise as a general-purpose non-parametric system for embodied agents.

📄 PDF Abstract BibTeX arXiv:2409.18313

Code (0)

등록된 구현이 없습니다.

Tasks

RAG RetrievalRetrieval-augmented GenerationText Generation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

WordPiece 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

BERT BERT, or Bidirectional Encoder Representations from Transformers, improves upon standard Transformers by removing the…

PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

2026-05-26 · Yuchen Guo, Junli Gong, Weicheng Wang, Hongmin Cai 외 arxiv

We present PEAM, a Parametric Embodied Agent Memory framework in Minecraft that transforms agent memory from inference-time retrieval into parameter-resident skills internalized through experience. PEAM pairs a slow deli…

Continual Learning

HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering

2026-02-17 · Ji Li, Bo Wang, Jing Xia, Mingyi Li 외 arxiv

Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on tex…

Analytic Concept-Centric Memory for Agentic Embodied Manipulation

2026-06-29 · Mingyang Sun, Xiujian Liang, Jiude Wei, Qichen He 외 arxiv

Long-horizon embodied manipulation requires agents to remember persistent objects, track changing scene states, and reuse prior interaction knowledge. However, existing agent memories are often stored as unstructured his…

Skill Generalization

EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents

2026-04-20 · Paolo Riva, Leonardo Gargani, Matteo Frosi, Matteo Matteucci arxiv

As the world of agentic artificial intelligence applied to robotics evolves, the need for agents capable of building and retrieving memories and observations efficiently is increasing. Robots operating in complex environ…

WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents

2026-06-17 · Yehang Zhang, Jianchong Su, Haojian Huang, Yifan Chang 외 arxiv

To assist humans over extended periods in real homes, embodied agents must remember user routines, world states, and past interactions. Existing long-term memory benchmarks mainly evaluate language-centric retrieval and …

Question Answering