ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
Large language models~(LLMs) have been adopted to process textual task description and accomplish procedural planning in embodied AI tasks because of their powerful reasoning ability. However, there is still lack of study on how vision language models~(VLMs) behave when multi-modal task inputs are considered. Counterfactual planning that evaluates the model's reasoning ability over alternative task situations are also under exploited. In order to evaluate the planning ability of both multi-modal and counterfactual aspects, we propose ActPlan-1K. ActPlan-1K is a multi-modal planning benchmark constructed based on ChatGPT and household activity simulator iGibson2. The benchmark consists of 153 activities and 1,187 instances. Each instance describing one activity has a natural language task description and multiple environment images from the simulator. The gold plan of each instance is action sequences over the objects in provided scenes. Both the correctness and commonsense satisfaction are evaluated on typical VLMs. It turns out that current VLMs are still struggling at generating human-level procedural plans for both normal activities and counterfactual activities. We further provide automatic evaluation metrics by finetuning over BLEURT model to facilitate future research on our benchmark.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingcounterfactualCounterfactual PlanningSimilar Papers 제목 키워드 기반
LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning
While large language models (LLMs) have advanced procedural planning for embodied AI systems through strong reasoning abilities, the integration of multimodal inputs and counterfactual reasoning remains underexplored. To…
GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning
Implicit spatial relations and deep semantic structures encoded in object attributes are crucial for procedural planning in embodied AI systems. However, existing approaches often over rely on the reasoning capabilities …
Contrastive LearningInFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers
The rapid growth of LLM inference is shifting sustainability concerns from one-time training to continuous serving, where infrastructure decisions shape energy use, carbon emissions, water consumption, and service qualit…
Procedural Generalization by Planning with Self-Supervised World Models
One of the key promises of model-based reinforcement learning is the ability to generalize using an internal model of the world to make predictions in novel environments and tasks. However, the generalization ability of …
BenchmarkingMeta-LearningModel-based Reinforcement LearningRepresentation LearningVG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting
Large Language Model (LLM)-based agents have shown promise in procedural tasks, but the potential of multimodal instructions augmented by texts and videos to assist users remains under-explored. To address this gap, we p…
InformativenessLarge Language ModelText GenerationText-to-Video Generation+2