ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
Graphical User Interface (GUI) agents are autonomous systems that interpret and generate actions, enabling intelligent user assistance and automation. Effective training of these agent presents unique challenges, such as sparsity in supervision signals, scalability for large datasets, and the need for nuanced user understanding. We propose stateful screen schema, an efficient representation of GUI interactions that captures key user actions and intentions over time. Building on this foundation, we introduce ScreenLLM, a set of multimodal large language models (MLLMs) tailored for advanced UI understanding and action prediction. Extensive experiments on both open-source and proprietary models show that ScreenLLM accurately models user behavior and predicts actions. Our work lays the foundation for scalable, robust, and intelligent GUI agents that enhance user interaction in diverse software environments.
Code (0)
등록된 구현이 없습니다.
Tasks
Action UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction
Persistent AI memory is often reduced to a retrieval problem: store prior interactions as text, embed them, and ask the model to recover relevant context later. This design is useful for thematic recall, but it is mismat…
Object DetectionLearning to Represent State with Perceptual Schemata
The real world is large and complex. It is filled with many objects besides those defined by a task and objects can move with their own interesting dynamics. How should an agent learn to represent state to support effici…
Zero-shot GeneralizationDiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context
Scientific papers use schematic diagrams to communicate methods, workflows, and system structure, yet existing scientific-figure corpora often mix them with plots, screenshots, and photographs and rarely preserve documen…
VetClaw: An Edge-Cloud Multimodal Agentic System for Veterinary Disease Screening
We present VetClaw, an edge-cloud multimodal agentic system for early veterinary disease screening. VetClaw uses a camera module as an edge sensing device and sends captured images, together with optional symptom descrip…
Image ClassificationExploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
As in many other different fields, deep learning has become the main approach in most computer vision applications, such as scene understanding, object recognition, computer-human interaction or human action recognition …
Action RecognitionObject RecognitionScene UnderstandingTemporal Action Localization