paper-with-me

Papers

A Zero-Shot Language Agent for Computer Control with Structured Reflection

2023-10-12 · Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, Yang Li

Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal in a live computer environment (e.g. MiniWoB++). To perform a task, recent works often require a model to learn from trace examples of the task via either supervised learning or few/many-shot prompting. Without these trace examples, it remains a challenge how an agent can autonomously learn and improve its control on a computer, which limits the ability of an agent to perform a new task. We approach this problem with a zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iteratively progresses a task by identifying and learning from its mistakes via self-reflection and structured thought management. On the easy tasks of MiniWoB++, we show that our zero-shot agent often outperforms recent SoTAs, with more efficient reasoning. For tasks with more complexity, our reflective agent performs on par with prior best models, even though previous works had the advantages of accessing expert traces or additional screen information.

📄 PDF Abstract BibTeX arXiv:2310.08740

Code (0)

등록된 구현이 없습니다.

Tasks

Management

Similar Papers 제목 키워드 기반

ScreenAgent: A Vision Language Model-driven Computer Control Agent

2024-02-09 · Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu 외

Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agen…

Language ModelingLanguage Modelling

Temporal UI State Inconsistency in Desktop GUI Agents: Formalizing and Defending Against TOCTOU Attacks on Computer-Use Agents

2026-04-20 · Wenpeng Xu arxiv

GUI agents that control desktop computers via screenshot-and-click loops introduce a new class of vulnerability: the observation-to-action gap (mean 6.51 s on real OSWorld workloads) creates a Time-Of-Check, Time-Of-Use …

Retcon -- a Prompt-Based Technique for Precise Control of LLMs in Conversations

2026-02-09 · David Kogan, Sam Nguyen, Masanori Suzuki, Feiyang Chen arxiv

Recent advances in Large Language Models (LLMs) allow agents to execute complex natural language tasks. Many LLM applications, such as support agents, teaching assistants, and interactive bots, involve multi-turn convers…

Agent-Computer Observation Interfaces Enable Dynamic Computer Use

2026-06-28 · Bojie Li, Noah Shi arxiv

SWE-agent established the action interface as an underexplored design axis for software-engineering agents; we make the analogous case for the observation interface in computer-use (CU) agents. Current CU agents, closed …

Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

2026-06-23 · Marta Sumyk, Oleksandr Kosovan arxiv

Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces. However, reinforcement learning for CUAs remains difficult because open-ended desktop environme…

Reinforcement Learning