Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development in which multimodal large language models (MLLMs) directly convert visual designs into code implementations. In this work, we construct Design2Code - the first real-world benchmark for this task. Specifically, we manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations to validate the performance ranking. To rigorously benchmark MLLMs, we test various multimodal prompting methods on frontier models such as GPT-4o, GPT-4V, Gemini, and Claude. Our fine-grained break-down metrics indicate that models mostly lag in recalling visual elements from the input webpages and generating correct layout designs.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingCode GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit
Multimodal Variational Autoencoders (VAEs) have been the subject of intense research in the past years as they can integrate multiple modalities into a joint representation and can thus serve as a promising tool for both…
BenchmarkingBenchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
Recent advancements in multimodal large language models (MLLMs) have achieved remarkable progress in multimodal reasoning and code generation, catalyzing a new paradigm for front-end development. In particular, these mod…
Multimodal ReasoningCode GenerationUnison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabili…
WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Automatically generating webpage code from webpage designs can significantly reduce the workload of front-end developers, and recent Multimodal Large Language Models (MLLMs) have shown promising potential in this area. H…
BenchmarkingCode GenerationFrom Intent to Execution: Multimodal Chain-of-Thought Reinforcement Learning for Precise CAD Code Generation
Computer-Aided Design (CAD) plays a vital role in engineering and manufacturing, yet current CAD workflows require extensive domain expertise and manual modeling effort. Recent advances in large language models (LLMs) ha…
Reinforcement LearningLogical ReasoningCode Generation