Where is Misty? Interpreting Spatial Descriptors by Modeling Regions in Space
We present a model for locating regions in space based on natural language descriptions. Starting with a 3D scene and a sentence, our model is able to associate words in the sentence with regions in the scene, interpret relations such as {}on top of{'} or {}next to,{'} and finally locate the region described in the sentence. All components form a single neural network that is trained end-to-end without prior knowledge of object segmentation. To evaluate our model, we construct and release a new dataset consisting of Minecraft scenes with crowdsourced natural language descriptions. We achieve a 32{\%} relative error reduction compared to a strong neural baseline.
Code (1)
Tasks
MinecraftSemantic SegmentationSentenceSimilar Papers 제목 키워드 기반
MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robots
With the availability of open APIs in social robots, it has become easier to customize general-purpose tools to meet users' needs. However, interpreting high-level user instructions, selecting and configuring appropriate…
AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot
The social robot's open API allows users to customize open-domain interactions. However, it remains inaccessible to those without programming experience. In this work, we introduce AutoMisty, the first multi-agent collab…
Code GenerationMISTY: High-Throughput Motion Planning via Mixer-based Single-step Drifting
Multi-modal trajectory generation is essential for safe autonomous driving, yet existing diffusion-based planners suffer from high inference latency due to iterative neural function evaluations. This paper presents MISTY…
Autonomous DrivingMotion PlanningHahahahaha, Duuuuude, Yeeessss!: A two-parameter characterization of stretchable words and the dynamics of mistypings and misspellings
Stretched words like `heellllp' or `heyyyyy' are a regular feature of spoken language, often used to emphasize or exaggerate the underlying meaning of the root word. While stretched words are rarely found in formal writt…
Comparing roughness maps generated by five roughness descriptors for LiDAR-derived digital elevation models
Terrain surface roughness, often described abstractly, poses challenges in quantitative characterisation with various descriptors found in the literature. This study compares five commonly used roughness descriptors, exp…