UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments
Developing and testing user interfaces (UIs) and training AI agents to interact with them are challenging due to the dynamic and diverse nature of real-world mobile environments. Existing methods often rely on cumbersome physical devices or limited static analysis of screenshots, which hinders scalable testing and the development of intelligent UI agents. We introduce UISim, a novel image-based UI simulator that offers a dynamic and interactive platform for exploring mobile phone environments purely from screen images. Our system employs a two-stage method: given an initial phone screen image and a user action, it first predicts the abstract layout of the next UI state, then synthesizes a new, visually consistent image based on this predicted layout. This approach enables the realistic simulation of UI transitions. UISim provides immediate practical benefits for UI testing, rapid prototyping, and synthetic data generation. Furthermore, its interactive capabilities pave the way for advanced applications, such as UI navigation task planning for AI agents. Our experimental results show that UISim outperforms end-to-end UI generation baselines in generating realistic and coherent subsequent UI states, highlighting its fidelity and potential to streamline UI development and enhance AI agent training.
Code (0)
등록된 구현이 없습니다.
Tasks
Synthetic Data GenerationSimilar Papers 제목 키워드 기반
SARDINE: A Simulator for Automated Recommendation in Dynamic and Interactive Environments
Simulators can provide valuable insights for researchers and practitioners who wish to improve recommender systems, because they allow one to easily tweak the experimental setup in which recommender systems operate, and …
counterfactualLearning-To-RankRecommendation SystemsRegTraffic: A Regression Based Traffic Simulator for Spatiotemporal Traffic Modeling, Simulation and Visualization
Traffic simulation is a great tool to demonstrate complex traffic structures which can be extremely useful for the planning, development, and management of road traffic networks. Current traffic simulators offer limited …
ManagementregressionInteractive World Simulator for Robot Policy Training and Evaluation
Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent i…
Video PredictionDeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality
We present a learning-based method to infer plausible high dynamic range (HDR), omnidirectional illumination given an unconstrained, low dynamic range (LDR) image from a mobile phone camera with a limited field of view (…
Mixed RealityJoint Learning of Interactive Spoken Content Retrieval and Trainable User Simulator
User-machine interaction is crucial for information retrieval, especially for spoken content retrieval, because spoken content is difficult to browse, and speech recognition has a high degree of uncertainty. In interacti…
Information RetrievalQ-LearningRetrievalspeech-recognition+1