paper-with-me

홈 › Papers

UI Layout Generation with LLMs Guided by UI Grammar

2023-10-24 · Yuwen Lu, Ziang Tong, Qinyi Zhao, Chengzhi Zhang, Toby Jia-Jun Li

The recent advances in Large Language Models (LLMs) have stimulated interest among researchers and industry professionals, particularly in their application to tasks concerning mobile user interfaces (UIs). This position paper investigates the use of LLMs for UI layout generation. Central to our exploration is the introduction of UI grammar -- a novel approach we proposed to represent the hierarchical structure inherent in UI screens. The aim of this approach is to guide the generative capacities of LLMs more effectively and improve the explainability and controllability of the process. Initial experiments conducted with GPT-4 showed the promising capability of LLMs to produce high-quality user interfaces via in-context learning. Furthermore, our preliminary comparative study suggested the potential of the grammar-based approach in improving the quality of generative results in specific aspects.

📄 PDF Abstract BibTeX arXiv:2310.15455

Code (0)

등록된 구현이 없습니다.

Tasks

In-Context LearningLayout GenerationPosition

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음
Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…
Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…
Adam 설명 없음
Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Similar Papers 제목 키워드 기반

Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedback

2023-05-25 · Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu 외

Generating and editing a 3D scene guided by natural language poses a challenge, primarily due to the complexity of specifying the positional relations and volumetric changes within the 3D space. Recent advancements in La…

3D Generation

GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting

2024-02-11 · Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He 외

We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. We first utilize large language models (LLMs) to generate the initial layout and introduce a layou…

3D GenerationScene GenerationText to 3D

VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning

2023-09-26 · Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal

Recent text-to-video (T2V) generation methods have seen significant advancements. However, the majority of these works focus on producing short video clips of a single event (i.e., single-scene videos). Meanwhile, recent…

Image GenerationVideo Generation

AI Coders Are Among Us: Rethinking Programming Language Grammar Towards Efficient Code Generation

2024-04-25 · Zhensu Sun, Xiaoning Du, Zhou Yang, Li Li 외

Artificial Intelligence (AI) models have emerged as another important audience for programming languages alongside humans and machines, as we enter the era of large language models (LLMs). LLMs can now perform well in co…

Code GenerationMath

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation

2023-04-13 · Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan 외

Spatial control is a core capability in controllable image generation. Advancements in layout-guided image generation have shown promising results on in-distribution (ID) datasets with similar spatial configurations. How…

DiagnosticImage GenerationLayout-to-Image Generation