UIPro: Unleashing Superior Interaction Capability For GUI Agents
Building autonomous agents that perceive and operate graphical user interfaces (GUIs) like humans has long been a vision in the field of artificial intelligence. Central to these agents is the capability for GUI interaction, which involves GUI understanding and planning capabilities. Existing methods have tried developing GUI agents based on the multi-modal comprehension ability of vision-language models (VLMs). However, the limited scenario, insufficient size, and heterogeneous action spaces hinder the progress of building generalist GUI agents. To resolve these issues, this paper proposes \textbf{UIPro}, a novel generalist GUI agent trained with extensive multi-platform and multi-task GUI interaction data, coupled with a unified action space. We first curate a comprehensive dataset encompassing 20.6 million GUI understanding tasks to pre-train UIPro, granting it a strong GUI grounding capability, which is key to downstream GUI agent tasks. Subsequently, we establish a unified action space to harmonize heterogeneous GUI agent task datasets and produce a merged dataset to foster the action prediction ability of UIPro via continued fine-tuning. Experimental results demonstrate UIPro's superior performance across multiple GUI task benchmarks on various platforms, highlighting the effectiveness of our approach.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Equiprobable mappings in weighted constraint grammars
We show that MaxEnt is so rich that it can distinguish between any two different mappings: there always exists a nonnegative weight vector which assigns them different MaxEnt probabilities. Stochastic HG instead does adm…
Dictionary Learning with Equiprobable Matching Pursuit
Sparse signal representations based on linear combinations of learned atoms have been used to obtain state-of-the-art results in several practical signal processing applications. Approximation methods are needed to proce…
DenoisingDictionary LearningA Variational Approach to Parameter Estimation for Characterizing 2-D Cluster Variation Method Topographies
One of the biggest challenges in characterizing 2-D topographies is succinctly communicating the dominant nature of local configurations. In a 2-D grid composed of bistate units, this could be expressed as finding the ch…
parameter estimationTripletVariational InferenceNavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
Capitalizing on the remarkable advancements in Large Language Models (LLMs), there is a burgeoning initiative to harness LLMs for instruction following robotic navigation. Such a trend underscores the potential of LLMs t…
Instruction FollowingVision and Language NavigationFree Energy Minimization Using the 2-D Cluster Variation Method: Initial Code Verification and Validation
A new approach for general artificial intelligence (GAI), building on neural network deep learning architectures, can make use of one or more hidden layers that have the ability to continuously reach a free energy minimu…