Faithful Mobile GUI Agents with Guided Advantage Estimator
Vision-language model based graphical user interface (GUI) agents have shown strong interaction capabilities. However, they often behave unfaithfully, relying on memorized shortcuts rather than grounding actions in displayed screen evidence or user instructions. To address this, we propose Faithful-Agent, a faithfulness-first framework that reformulates GUI interaction to prioritize evidence groundedness and internal consistency. Faithful-Agent employs a two-stage pipeline: (i) a faithfulness-oriented SFT stage to instill abstainment behaviors under evidence perturbations; (ii) an RFT stage that further amplifies faithfulness by introducing the guided advantage estimator (GuAE), an anchor-based and variance-adaptive advantage tempering mechanism built upon GRPO. GuAE prevents advantage collapse in low-variance rollout groups under sparse GUI rewards, and with a thought-action consistency reward, Faithful-Agent (Stage II) elevates the Trap SR from 13.88\% to 80.21\% relative to the baseline, while preserving robust general instruction-following performance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
The emergent large language/multimodal models facilitate the evolution of mobile agents, especially in mobile UI task automation. However, existing evaluation approaches, which rely on human validation or established dat…
DocOS: Towards Proactive Document-Guided Actions in GUI Agents
While Graphical User Interface (GUI) agents have shown promising performance in automated device interaction, they primarily depend on static parametric knowledge from pre-training or instruction tuning. This reliance fu…
Faithful Edge Federated Learning: Scalability and Privacy
Federated learning enables machine learning algorithms to be trained over a network of multiple decentralized edge devices without requiring the exchange of local datasets. Successfully deploying federated learning requi…
Federated LearningRange-Only Bearing Estimator for Localization and Mapping
Navigation and exploration within unknown environments are typical examples in which simultaneous localization and mapping (SLAM) algorithms are applied. When mobile agents deploy only range sensors without bearing infor…
Simultaneous Localization and MappingBiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents
Stepwise group-based RL is an attractive way to train long-horizon LLM agents without a learned critic: it reuses multiple sampled rollouts to estimate local advantages. Its weakness is less visible but more fundamental:…