Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents
Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$\,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.
Code (0)
등록된 구현이 없습니다.
Results from the Paper
| Rank | Task | Dataset | Model | Metrics |
|---|---|---|---|---|
| #9 | Natural Language Queries | Ego4D | Linguistic Trajectory Encoding for Effic | R@1 Mean(0.3 and 0.5): 8.7 |
Similar Papers 제목 키워드 기반
State Design Matters: How Representations Shape Dynamic Reasoning in Large Language Models
As large language models (LLMs) move from static reasoning tasks toward dynamic environments, their success depends on the ability to navigate and respond to an environment that changes as they interact at inference time…
Spatial ReasoningTrajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting
Multi-person pose forecasting remains a challenging problem, especially in modeling fine-grained human body interaction in complex crowd scenarios. Existing methods typically represent the whole pose sequence as a tempor…
Multi-Person Pose forecastingLongFly: Long-Horizon UAV Vision-and-Language Navigation with Spatiotemporal Context Integration
Unmanned aerial vehicles (UAVs) are crucial tools for post-disaster search and rescue, facing challenges such as high information density, rapid changes in viewpoint, and dynamic structures, especially in long-horizon na…
Image CompressionSTF: Spatial Temporal Fusion for Trajectory Prediction
Trajectory prediction is a challenging task that aims to predict the future trajectory of vehicles or pedestrians over a short time horizon based on their historical positions. The main reason is that the trajectory is a…
Graph AttentionPredictionTrajectory PredictionQuery-Centric Trajectory Prediction
Predicting the future trajectories of surrounding agents is essential for autonomous vehicles to operate safely. This paper presents QCNet, a modeling framework toward pushing the boundaries of trajectory prediction.…
Autonomous DrivingAutonomous VehiclesMotion Forecastingmotion prediction+3