UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior
With the fast-growing GUI development workload in the Internet industry, some work on intelligent methods attempted to generate maintainable front-end code from UI screenshots. It can be more suitable for utilizing UI design drafts that contain UI metadata. However, fragmented layers inevitably appear in the UI design drafts which greatly reduces the quality of code generation. None of the existing GUI automated techniques detects and merges the fragmented layers to improve the accessibility of generated code. In this paper, we propose UI Layers Merger (UILM), a vision-based method, which can automatically detect and merge fragmented layers into UI components. Our UILM contains Merging Area Detector (MAD) and a layers merging algorithm. MAD incorporates the boundary prior knowledge to accurately detect the boundaries of UI components. Then, the layers merging algorithm can search out the associated layers within the components' boundaries and merge them into a whole part. We present a dynamic data augmentation approach to boost the performance of MAD. We also construct a large-scale UI dataset for training the MAD and testing the performance of UILM. The experiment shows that the proposed method outperforms the best baseline regarding merging area detection and achieves a decent accuracy regarding layers merging.
Code (1)
Tasks
Code GenerationData AugmentationSimilar Papers 제목 키워드 기반
LARV: Data-Free Layer-wise Adaptive Rescaling Veneer for Model Merging
Model merging aims to combine multiple fine-tuned models into a single multi-task model without access to training data. Existing task-vector merging methods such as TIES, TSV-M, and Iso-C/CTS differ in their aggregation…
Learning to Merge Tokens in Vision Transformers
Transformers are widely applied to solve natural language understanding and computer vision tasks. While scaling up these architectures leads to improved performance, it often comes at the expense of much higher computat…
Natural Language UnderstandingMergeRec: Model Merging for Data-Isolated Cross-Domain Sequential Recommendation
Modern recommender systems trained on domain-specific data often struggle to generalize across multiple domains. Cross-domain sequential recommendation has emerged as a promising research direction to address this challe…
Sequential RecommendationCollaborative FilteringRevisiting the Impact of Upstream Mergers with Downstream Complements and Substitutes
I examine how upstream mergers affect negotiated prices when suppliers bargain with a monopoly intermediary selling products to final consumers. Conventional wisdom holds that such transactions lower negotiated prices wh…
Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribu…