Automatic code generation from sketches of mobile applications in end-user development using Deep Learning
A common need for mobile application development by end-users or in computing education is to transform a sketch of a user interface into wireframe code using App Inventor, a popular block-based programming environment. As this task is challenging and time-consuming, we present the Sketch2aia approach that automates this process. Sketch2aia employs deep learning to detect the most frequent user interface components and their position on a hand-drawn sketch creating an intermediate representation of the user interface and then automatically generates the App Inventor code of the wireframe. The approach achieves an average user interface component classification accuracy of 87,72% and results of a preliminary user evaluation indicate that it generates wireframes that closely mirror the sketches in terms of visual similarity. The approach has been implemented as a web tool and can be used to support the end-user development of mobile applications effectively and efficiently as well as the teaching of user interface design in K-12.
Code (1)
Tasks
Code GenerationComponent ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
BézierSketch: A generative model for scalable vector sketches
The study of neural generative models of human sketches is a fascinating contemporary modeling problem due to the links between sketch image generation and the human drawing process. The landmark SketchRNN provided break…
Image GenerationmodelCreative Sketch Generation
Sketching or doodling is a popular creative activity that people engage in. However, most existing work in automatic sketch understanding or generation has focused on sketches that are quite mundane. In this work, we int…
Generative Adversarial NetworkSketchyCOCO: Image Generation from Freehand Scene Sketches
We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribu…
AttributeGenerative Adversarial NetworkImage GenerationObject+1High-Quality Medical Image Generation from Free-hand Sketch
Generating medical images from human-drawn free-hand sketches holds promise for various important medical imaging applications. Due to the extreme difficulty in collecting free-hand sketch data in the medical domain, mos…
Image GenerationMedical Image GenerationAutomatic code generation from low fidelity graphical user interface sketches using deep learning
In software development, the user interface(UI) design process involves the part from an imagination that begins on a piece of paper where designers share thoughts to the final prototype that is ready to implement. UI de…
Code Generation