GPA: Learning GUI Process Automation from Demonstrations
GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
UINav: A Practical Approach to Train On-Device Automation Agents
Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired. Prior automation systems do not pro…
DiversityDemonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot
Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive …
Efficient Explorationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Deep learning in automated ultrasonic NDE -- developments, axioms and opportunities
The analysis of ultrasonic NDE data has traditionally been addressed by a trained operator manually interpreting data with the support of rudimentary automation tools. Recently, many demonstrations of deep learning (DL) …
Deep LearningDefect DetectionDiLogics: Creating Web Automation Programs With Diverse Logics
Knowledge workers frequently encounter repetitive web data entry tasks, like updating records or placing orders. Web automation increases productivity, but translating tasks to web actions accurately and extending to new…
BMW Agents -- A Framework For Task Automation Through Multi-Agent Collaboration
Autonomous agents driven by Large Language Models (LLMs) offer enormous potential for automation. Early proof of this technology can be found in various demonstrations of agents solving complex tasks, interacting with ex…