GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningVisual GroundingSimilar Papers 제목 키워드 기반
Learning Visual Features from Large Weakly Supervised Data
Convolutional networks trained on large supervised dataset produce visual features which form the basis for the state-of-the-art in many computer-vision problems. Further improvements of these visual features will likely…
Representation LearningWord SimilarityLeveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation
Weakly supervised semantic instance segmentation with only image-level supervision, instead of relying on expensive pixel wise masks or bounding box annotations, is an important problem to alleviate the data-hungry natur…
Image-level Supervised Instance SegmentationInstance SegmentationMultiple Instance LearningSegmentation+3Weakly Supervised Construction of ASR Systems with Massive Video Data
Building Automatic Speech Recognition (ASR) systems from scratch is significantly challenging, mostly due to the time-consuming and financially-expensive process of annotating a large amount of audio data with transcript…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Optical Character RecognitionOptical Character Recognition (OCR)+3Learning a Weakly-Supervised Video Actor-Action Segmentation Model with a Wise Selection
We address weakly-supervised video actor-action segmentation (VAAS), which extends general video object segmentation (VOS) to additionally consider action labels of the actors. The most successful methods on VOS synthesi…
Action SegmentationSegmentationSemantic SegmentationVideo Object Segmentation+1Agentic Spatio-Temporal Grounding via Collaborative Reasoning
Spatio-Temporal Video Grounding (STVG) aims to retrieve the spatio-temporal tube of a target object or person in a video given a text query. Most existing approaches perform frame-wise spatial localization within a predi…
Spatio-Temporal Video GroundingSpatial Reasoning