PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks
Unified multimodal models (UMMs) have shown impressive capabilities in generating natural images and supporting multimodal reasoning. However, their potential in supporting computer-use planning tasks, which are closely related to our lives, remain underexplored. Image generation and editing in computer-use tasks require capabilities like spatial reasoning and procedural understanding, and it is still unknown whether UMMs have these capabilities to finish these tasks or not. Therefore, we propose PlanViz, a new benchmark designed to evaluate image generation and editing for computer-use tasks. To achieve the goal of our evaluation, we focus on sub-tasks which frequently involve in daily life and require planning. Specifically, three representative sub-tasks are designed: route planning, work diagramming, and web&UI displaying. We address challenges in data quality ensuring by curating human-annotated questions and reference images, and a quality control process. For detailed and exact evaluation, a task-adaptive score, PlanScore, is proposed. The score helps understanding the correctness, visual quality and efficiency of generated images. Through experiments, we highlight key limitations and opportunities for future research on this topic.
Code (0)
등록된 구현이 없습니다.
Tasks
Multimodal ReasoningSpatial ReasoningImage GenerationSimilar Papers 제목 키워드 기반
Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos
Goal-oriented planning, or anticipating a series of actions that transition an agent from its current state to a predefined objective, is crucial for developing intelligent assistants aiding users in daily procedural tas…
TopKG: Target-oriented Dialog via Global Planning on Knowledge Graph
Target-oriented dialog aims to reach a global target through multi-turn conversation. The key to the task is the global planning towards the target, which flexibly guides the dialog concerning the context. However, exist…
Response GenerationTowards Evaluating Plan Generation Approaches with Instructional Texts
Recent research in behaviour understanding through language grounding has shown it is possible to automatically generate behaviour models from textual instructions. These models usually have goal-oriented structure and a…
SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency
Recent text-to-image models have revolutionized image generation, but they still struggle with maintaining concept consistency across generated images. While existing works focus on character consistency, they often over…
Story GenerationImage GenerationTarget-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue
Target-oriented proactive dialogue systems aim to lead conversations from a dialogue context toward a pre-determined target, such as making recommendations on designated items or introducing new specific topics. To this …
Dialogue Generation